[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489489538 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003095] x2apic enabled [ 0.004018] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008434] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.022626] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026603] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029127] *** VALIDATE tmpfs *** [ 0.031398] *** VALIDATE proc *** [ 0.032238] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.035007] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041871] debug: unmapping init [mem 0xffffffff96259000-0xffffffff96260fff] [ 0.044165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045731] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053295] rcu: Hierarchical SRCU implementation. [ 0.055401] smp: Bringing up secondary CPUs ... [ 0.056515] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.060862] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132609] node 0 deferred pages initialised in 68ms [ 0.136007] devtmpfs: initialized [ 0.137273] x86/mm: Memory block size: 128MB [ 0.139744] gcov: version magic: 0x41383552 [ 0.143287] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150283] pinctrl core: initialized pinctrl subsystem [ 0.153199] [ 0.153794] ************************************************************* [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159014] ** ** [ 0.162013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164013] ** ** [ 0.167014] ** This means that this kernel is built to expose internal ** [ 0.170013] ** IOMMU data structures, which may compromise security on ** [ 0.173014] ** your system. ** [ 0.176026] ** ** [ 0.178012] ** If you see this message and you are not debugging the ** [ 0.181015] ** kernel, report this immediately to your vendor! ** [ 0.184013] ** ** [ 0.187014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189012] ************************************************************* [ 0.192635] NET: Registered protocol family 16 [ 0.194468] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205016] cpuidle: using governor menu [ 0.206985] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210378] PCI: Using configuration type 1 for base access [ 0.212117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223127] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226059] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230048] cryptd: max_cpu_qlen set to 1000 [ 0.233229] ACPI: Added _OSI(Module Device) [ 0.234011] ACPI: Added _OSI(Processor Device) [ 0.236013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244126] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251201] ACPI: Interpreter enabled [ 0.254070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255012] ACPI: Using IOAPIC for interrupt routing [ 0.257094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.261355] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.275041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.278018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.290000] acpiphp: Slot [2] registered [ 0.292197] acpiphp: Slot [5] registered [ 0.294163] acpiphp: Slot [6] registered [ 0.295171] acpiphp: Slot [7] registered [ 0.298027] acpiphp: Slot [8] registered [ 0.300140] acpiphp: Slot [9] registered [ 0.301111] acpiphp: Slot [10] registered [ 0.303129] acpiphp: Slot [3] registered [ 0.304091] acpiphp: Slot [4] registered [ 0.306132] acpiphp: Slot [11] registered [ 0.307103] acpiphp: Slot [12] registered [ 0.310210] acpiphp: Slot [13] registered [ 0.311117] acpiphp: Slot [14] registered [ 0.313107] acpiphp: Slot [15] registered [ 0.315190] acpiphp: Slot [16] registered [ 0.316156] acpiphp: Slot [17] registered [ 0.318107] acpiphp: Slot [18] registered [ 0.320103] acpiphp: Slot [19] registered [ 0.321143] acpiphp: Slot [20] registered [ 0.323106] acpiphp: Slot [21] registered [ 0.324109] acpiphp: Slot [22] registered [ 0.325115] acpiphp: Slot [23] registered [ 0.327114] acpiphp: Slot [24] registered [ 0.328105] acpiphp: Slot [25] registered [ 0.330096] acpiphp: Slot [26] registered [ 0.331096] acpiphp: Slot [27] registered [ 0.333115] acpiphp: Slot [28] registered [ 0.335119] acpiphp: Slot [29] registered [ 0.337119] acpiphp: Slot [30] registered [ 0.338108] acpiphp: Slot [31] registered [ 0.340069] PCI host bridge to bus 0000:00 [ 0.342025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.344018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.346026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.349023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.352025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.356026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.361770] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365426] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.375014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.381014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.388014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.390618] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.394106] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.397040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.400811] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.407014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.421015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.427018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435513] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.443015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.465076] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.477206] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.486014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.525024] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.533020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.544017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.562015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.572174] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.581032] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.588026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.605025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.616231] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.622019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.629016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.649014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.658542] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.664021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.670022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.683028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.696043] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.699486] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.703085] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.706390] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.708219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.712127] iommu: Default domain type: Passthrough [ 0.714525] SCSI subsystem initialized [ 0.716214] ACPI: bus type USB registered [ 0.718129] usbcore: registered new interface driver usbfs [ 0.720136] usbcore: registered new interface driver hub [ 0.722156] usbcore: registered new device driver usb [ 0.724222] pps_core: LinuxPPS API ver. 1 registered [ 0.726019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.730090] PTP clock support registered [ 0.733116] EDAC MC: Ver: 3.0.0 [ 0.735027] PCI: Using ACPI for IRQ routing [ 0.736879] NetLabel: Initializing [ 0.737010] NetLabel: domain hash size = 128 [ 0.738010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.739187] NetLabel: unlabeled traffic allowed by default [ 0.741169] vgaarb: loaded [ 0.743323] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.745014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.751686] clocksource: Switched to clocksource kvm-clock [ 0.857428] VFS: Disk quotas dquot_6.6.0 [ 0.859203] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.861984] *** VALIDATE ramfs *** [ 0.863701] *** VALIDATE hugetlbfs *** [ 0.865335] pnp: PnP ACPI init [ 0.868227] pnp: PnP ACPI: found 6 devices [ 0.887741] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.892264] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.895487] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.898188] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.900934] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.903904] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.906777] NET: Registered protocol family 2 [ 0.909742] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.917902] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.922077] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.928090] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.931805] TCP: Hash tables configured (established 65536 bind 65536) [ 0.934641] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.938108] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941597] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944451] NET: Registered protocol family 1 [ 0.947598] RPC: Registered named UNIX socket transport module. [ 0.950388] RPC: Registered udp transport module. [ 0.952243] RPC: Registered tcp transport module. [ 0.954072] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.956863] NET: Registered protocol family 44 [ 0.958343] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.960921] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.963625] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.966487] PCI: CLS 0 bytes, default 64 [ 0.968600] Unpacking initramfs... [ 2.415305] debug: unmapping init [mem 0xffff9eb2bcc54000-0xffff9eb2bffbffff] [ 2.419527] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.421877] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.424911] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.965284] Initialise system trusted keyrings [ 2.966691] Key type blacklist registered [ 2.969872] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.995947] zbud: loaded [ 2.999954] *** VALIDATE nfs *** [ 3.001464] *** VALIDATE nfs4 *** [ 3.003391] pstore: using deflate compression [ 3.009896] Platform Keyring initialized [ 3.130372] NET: Registered protocol family 38 [ 3.132481] Key type asymmetric registered [ 3.134277] Asymmetric key parser 'x509' registered [ 3.136602] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.140187] io scheduler mq-deadline registered [ 3.142275] io scheduler kyber registered [ 3.144239] io scheduler bfq registered [ 3.146380] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.149947] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.153262] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.156695] ACPI: Power Button [PWRF] [ 3.163362] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.170565] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.183657] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.191472] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.212578] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.243032] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.273270] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.279680] Non-volatile memory driver v1.3 [ 3.281662] Linux agpgart interface v0.103 [ 3.319330] virtio_blk virtio1: [vda] 145808 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.323482] vda: detected capacity change from 0 to 74653696 [ 3.337085] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.339824] vdb: detected capacity change from 0 to 1073741824 [ 3.355423] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.359121] vdc: detected capacity change from 0 to 2621440000 [ 3.374853] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.378172] vdd: detected capacity change from 0 to 2621440000 [ 3.398494] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.402364] vde: detected capacity change from 0 to 4294967296 [ 3.420894] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.424473] vdf: detected capacity change from 0 to 4294967296 [ 3.431692] libphy: Fixed MDIO Bus: probed [ 3.436861] usbcore: registered new interface driver usbserial_generic [ 3.439639] usbserial: USB Serial support registered for generic [ 3.442151] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.446553] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.448535] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.452033] mousedev: PS/2 mouse device common for all mice [ 3.455327] rtc_cmos 00:05: RTC can wake from S4 [ 3.458255] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.459271] rtc_cmos 00:05: registered as rtc0 [ 3.463902] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.464107] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.469298] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.470540] intel_pstate: CPU model not supported [ 3.478349] hid: raw HID events driver (C) Jiri Kosina [ 3.480831] usbcore: registered new interface driver usbhid [ 3.482853] usbhid: USB HID core driver [ 3.484757] drop_monitor: Initializing network drop monitor service [ 3.487603] Initializing XFRM netlink socket [ 3.489884] NET: Registered protocol family 10 [ 3.493496] Segment Routing with IPv6 [ 3.494844] NET: Registered protocol family 17 [ 3.496907] mpls_gso: MPLS GSO support [ 3.506214] RAS: Correctable Errors collector initialized. [ 3.509117] AVX version of gcm_enc/dec engaged. [ 3.510846] AES CTR mode by8 optimization enabled [ 3.590953] sched_clock: Marking stable (3590860508, 0)->(4544426641, -953566133) [ 3.595570] registered taskstats version 1 [ 3.597661] Loading compiled-in X.509 certificates [ 3.600250] zswap: loaded using pool lzo/zbud [ 3.626367] Key type big_key registered [ 3.640571] Key type encrypted registered [ 3.642217] ima: No TPM chip found, activating TPM-bypass! [ 3.644180] ima: Allocated hash algorithm: sha1 [ 3.645864] ima: No architecture policies found [ 3.647307] evm: Initialising EVM extended attributes: [ 3.648815] evm: security.selinux [ 3.649881] evm: security.ima [ 3.651225] evm: security.capability [ 3.652478] evm: HMAC attrs: 0x1 [ 3.654896] rtc_cmos 00:05: setting system clock to 2026-07-13 12:43:25 UTC (1783946605) [ 3.661465] debug: unmapping init [mem 0xffffffff97203000-0xffffffff973fffff] [ 3.664896] debug: unmapping init [mem 0xffffffff95f82000-0xffffffff96258fff] [ 3.673149] Write protecting the kernel read-only data: 28672k [ 3.676829] debug: unmapping init [mem 0xffffffff94603000-0xffffffff947fffff] [ 3.680399] debug: unmapping init [mem 0xffffffff94f14000-0xffffffff94ffffff] [ 3.722392] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.731684] systemd[1]: Detected virtualization kvm. [ 3.734118] systemd[1]: Detected architecture x86-64. [ 3.736658] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.767086] systemd[1]: No hostname configured. [ 3.769365] systemd[1]: Set hostname to . [ 3.771788] random: systemd: uninitialized urandom read (16 bytes read) [ 3.774606] systemd[1]: Initializing machine ID from random generator. [ 3.823610] random: ln: uninitialized urandom read (6 bytes read) [ 3.913285] random: systemd: uninitialized urandom read (16 bytes read) [ 3.916326] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.921880] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.926170] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.561193] device-mapper: uevent: version 1.0.3 [ 4.563483] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.211972] virtio_net virtio0 ens2: renamed from eth0 [ 5.250401] scsi host0: ata_piix [ 5.262723] random: fast init done [ 5.264340] scsi host1: ata_piix [ 5.267123] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.271293] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.455969] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.157279] random: crng init done [ 10.158869] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.421960] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.474988] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.712905] SELinux: Disabled at runtime. [ 11.774717] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.785688] systemd[1]: Detected virtualization kvm. [ 11.787975] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.240413] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.244516] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.249610] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.255218] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.259831] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.266813] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.271193] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.373369] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.634609] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.877538] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.878807] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.024211] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.044199] EDAC sbridge: Ver: 1.1.2 [ 14.588172] Key type dns_resolver registered [ 14.904689] NFS: Registering the id_resolver key type [ 14.907465] Key type id_resolver registered [ 14.909383] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg309-server login: [ 40.790749] libcfs: loading out-of-tree module taints kernel. [ 40.811677] Key type ._llcrypt registered [ 40.813566] Key type .llcrypt registered [ 40.867796] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 49.997821] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 50.696725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.734681] alg: No test for adler32 (adler32-zlib) [ 51.805310] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 52.202865] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 53.848215] Key type lgssc registered [ 54.767118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.679825] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 119.146799] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 129.911007] hrtimer: interrupt took 7996735 ns [ 133.366260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 133.388759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.622163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 134.647648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 134.730794] Lustre: lustre-MDT0000: new disk, initializing [ 134.876371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.892534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.403485] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.059092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.199970] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 155.229957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 155.234680] Lustre: Skipped 1 previous similar message [ 155.303124] Lustre: lustre-MDT0001: new disk, initializing [ 155.358182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 155.382328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 155.395510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 160.593818] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.492603] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.758815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.259745] Lustre: lustre-OST0000: new disk, initializing [ 176.267519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.275748] Lustre: 8422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 176.415690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.114415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 178.136653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 178.277236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 183.398598] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.607466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.780387] Lustre: lustre-OST0001: new disk, initializing [ 198.792240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 198.803585] Lustre: 9495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.897706] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.255832] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.438131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.449141] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 207.509539] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 218.754539] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.819803] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 231.472609] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.127710] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing yml_node [ 241.921130] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 244.705968] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 247.533420] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 249.853891] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 13 08:47:28 EDT 2026 [ 290.478948] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 292.114857] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 309.217042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 309.240638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.270476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.313326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.315981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.325113] Lustre: Skipped 1 previous similar message [ 313.338194] Lustre: Skipped 3 previous similar messages [ 315.194749] Lustre: server umount lustre-MDT0000 complete [ 319.148798] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783946920 with bad export cookie 6256971480383812676 [ 319.149025] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.159703] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 323.553267] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.569972] Lustre: Skipped 2 previous similar messages [ 323.573505] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.577778] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.605675] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 324.577350] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.599712] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 325.531859] Lustre: server umount lustre-MDT0001 complete [ 330.315918] Lustre: server umount lustre-OST0000 complete [ 334.547871] Lustre: server umount lustre-OST0001 complete [ 342.043351] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 353.265397] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 396.327384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.628918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.665491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.778964] Lustre: lustre-MDT0000: new disk, initializing [ 396.866495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.900128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.756827] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 413.441074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.640942] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 413.742824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 413.750222] Lustre: Skipped 1 previous similar message [ 413.910226] Lustre: lustre-MDT0001: new disk, initializing [ 414.073639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 414.172220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 414.178725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 419.163660] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 429.985152] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 436.482349] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 443.512556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.944154] Lustre: lustre-OST0000: new disk, initializing [ 443.950194] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 443.964753] Lustre: 17041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 444.104446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.269603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 445.285646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 445.441370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 451.456395] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 462.027319] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 467.489147] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.839418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 474.635652] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.057941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 481.254498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 481.257064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.258341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.258348] Lustre: Skipped 3 previous similar messages [ 481.269964] LustreError: Skipped 1 previous similar message [ 481.300849] Lustre: Skipped 1 previous similar message [ 483.791217] Lustre: server umount lustre-OST0000 complete [ 490.982273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 490.998946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.020918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 491.030460] Lustre: Skipped 1 previous similar message [ 494.448865] Lustre: server umount lustre-MDT0000 complete [ 496.610667] LustreError: 15079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.629828] LustreError: 15079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 498.821170] LustreError: 15864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947100 with bad export cookie 6256971480383814209 [ 498.825414] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 498.850680] LustreError: 15864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 499.337803] Lustre: server umount lustre-MDT0001 complete [ 506.640128] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:51:45 (1783947105) [ 527.952472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.400366] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.480091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 533.482313] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.724207] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 543.091884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.632413] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 557.907128] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 564.338121] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 567.109171] Lustre: Failing over lustre-MDT0000 [ 567.532503] Lustre: server umount lustre-MDT0000 complete [ 568.168675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.172125] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.182693] Lustre: Skipped 2 previous similar messages [ 568.197534] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 584.673819] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947170/real 1783947170] req@ffff9eb3052d0380 x1870603646838144/t0(0) o400->MGC192.168.203.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783947186 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 584.711193] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 584.737206] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.773282] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 590.625462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.497831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 595.515463] Lustre: Skipped 1 previous similar message [ 600.548849] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 600.565318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 600.587748] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 602.006527] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 613.526523] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 618.982412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.985759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 618.989607] Lustre: Skipped 1 previous similar message [ 619.004971] Lustre: Skipped 3 previous similar messages [ 632.288149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 632.435311] Lustre: server umount lustre-MDT0000 complete [ 634.353574] LustreError: 21082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.376713] LustreError: 21082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 636.963674] LustreError: 19899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947238 with bad export cookie 6256971480383815623 [ 636.981353] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.983788] LustreError: 19899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 637.779567] Lustre: server umount lustre-MDT0001 complete [ 649.189607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.948651] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 664.223212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.546261] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 664.561154] Lustre: Skipped 4 previous similar messages [ 670.166129] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 679.249992] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 686.490852] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 688.963140] Lustre: Failing over lustre-MDT0000 [ 689.289171] Lustre: server umount lustre-MDT0000 complete [ 689.648198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.674770] Lustre: Skipped 1 previous similar message [ 700.903127] LustreError: 23267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.933664] LustreError: 23267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 706.018105] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947291/real 1783947291] req@ffff9eb207a5f100 x1870603646868736/t0(0) o400->MGC192.168.203.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783947307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 706.045831] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 709.537607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.693750] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 720.802944] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 720.866503] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 720.879475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 720.896167] Lustre: Skipped 1 previous similar message [ 720.928250] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 731.700983] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 740.323077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 740.330848] Lustre: Skipped 1 previous similar message [ 742.870950] Lustre: server umount lustre-MDT0000 complete [ 747.492071] LustreError: 23249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947349 with bad export cookie 6256971480383816561 [ 747.500402] LustreError: 23249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 747.502805] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 747.986499] Lustre: server umount lustre-MDT0001 complete [ 759.575806] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:55:58 (1783947358) [ 778.927984] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 789.904832] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 838.510304] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 852.012674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.330227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 852.381933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 852.570320] Lustre: lustre-MDT0000: new disk, initializing [ 852.688953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.692204] Lustre: Skipped 3 previous similar messages [ 852.733757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 858.101801] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 871.226673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.413247] Lustre: 29647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 871.920704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 877.661719] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 888.653828] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 895.800487] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 904.266651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 904.691531] Lustre: 31618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 906.378218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 906.420038] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 906.530048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 906.541573] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 912.162799] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 923.591607] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 930.195459] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 952.805167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.814226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 952.815094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 952.816210] Lustre: Skipped 3 previous similar messages [ 952.836215] LustreError: Skipped 1 previous similar message [ 957.340773] Lustre: server umount lustre-OST0000 complete [ 967.729937] Lustre: server umount lustre-MDT0000 complete [ 968.166638] LustreError: 29657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.195930] LustreError: 29657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 971.778447] LustreError: 31617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947573 with bad export cookie 6256971480383816967 [ 971.788922] LustreError: 31617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 971.790037] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.328403] Lustre: server umount lustre-MDT0001 complete [ 984.772866] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 987.866573] Key type lgssc unregistered [ 988.177793] LNet: 33595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.182060] LNetError: 33595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.204158] LNet: Removed LNI 192.168.203.109@tcp [ 989.290145] Key type .llcrypt unregistered [ 989.292700] Key type ._llcrypt unregistered [ 1013.747425] Key type ._llcrypt registered [ 1013.750516] Key type .llcrypt registered [ 1013.866758] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1015.390236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.427642] alg: No test for adler32 (adler32-zlib) [ 1016.588518] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 1016.925099] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1018.592159] Key type lgssc registered [ 1019.801581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1028.253366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1028.265854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.751654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1034.195285] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1039.332523] LustreError: 35008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.350603] LustreError: 35008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1043.788616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.298673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1049.545707] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1057.979784] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1064.496767] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1072.618756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1073.070215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.168332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1079.959283] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1087.763387] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1089.531685] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1102.343387] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:01:41 (1783947701) [ 1104.525343] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1106.345737] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1114.120912] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:01:53 (1783947713) [ 1123.332972] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:02:02 (1783947722) [ 1124.505065] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1133.557425] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1133.564209] Lustre: Skipped 2 previous similar messages [ 1135.203212] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:02:14 (1783947734) [ 1139.057165] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1139.065029] Lustre: Skipped 3 previous similar messages [ 1158.535910] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1158.541309] Lustre: Skipped 6 previous similar messages [ 1161.286604] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:02:40 (1783947760) [ 1164.162701] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1168.181501] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1177.433038] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:02:56 (1783947776) [ 1179.404559] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1179.410249] Lustre: Skipped 3 previous similar messages [ 1192.354599] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:03:11 (1783947791) [ 1207.266523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1207.279400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.285317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1207.318315] Lustre: Skipped 1 previous similar message [ 1210.823582] Lustre: server umount lustre-OST0000 complete [ 1217.506122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.528326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1217.531344] Lustre: Skipped 1 previous similar message [ 1220.836756] Lustre: server umount lustre-MDT0000 complete [ 1222.624887] LustreError: 35008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.638706] LustreError: 35008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1224.866794] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947826 with bad export cookie 4281428996155538645 [ 1224.875456] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1224.887693] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1225.495623] Lustre: server umount lustre-MDT0001 complete [ 1236.623502] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1239.904860] Key type lgssc unregistered [ 1240.263913] LNet: 42891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.287926] LNetError: 42891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.318314] LNet: Removed LNI 192.168.203.109@tcp [ 1241.534840] Key type .llcrypt unregistered [ 1241.539739] Key type ._llcrypt unregistered [ 1261.642110] Key type ._llcrypt registered [ 1261.643444] Key type .llcrypt registered [ 1261.719193] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 1277.569438] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1278.717662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1278.856089] alg: No test for adler32 (adler32-zlib) [ 1279.971486] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 1280.272748] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1281.945155] Key type lgssc registered [ 1283.075743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1318.181170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1318.205807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.350318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1319.379525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1319.442535] Lustre: lustre-MDT0000: new disk, initializing [ 1319.542743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.569281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1324.164393] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1334.972183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.127382] Lustre: 46048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1335.205578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1335.210273] Lustre: Skipped 1 previous similar message [ 1335.299813] Lustre: lustre-MDT0001: new disk, initializing [ 1335.389806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.426336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1335.435870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1340.022974] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1349.027784] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1355.538684] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1362.655428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1362.932671] Lustre: lustre-OST0000: new disk, initializing [ 1362.936275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1362.942492] Lustre: 48026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1363.068637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.177537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1368.195873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1368.419482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1369.946723] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1379.163452] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1383.148465] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1383.360275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1388.838138] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1389.048820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1394.147506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1394.159405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1394.170139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1399.285785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1399.301086] Lustre: Skipped 2 previous similar messages [ 1404.388972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1404.402614] Lustre: Skipped 1 previous similar message [ 1405.408144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1405.714439] Lustre: server umount lustre-OST0000 complete [ 1412.066203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1412.083072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.106975] Lustre: Skipped 1 previous similar message [ 1412.117378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.629875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.751980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1419.765371] Lustre: Skipped 3 previous similar messages [ 1424.352100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1424.578323] Lustre: server umount lustre-MDT0000 complete [ 1428.737711] LustreError: 46039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948030 with bad export cookie 10017444530756239744 [ 1428.741313] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1428.747501] LustreError: 46039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1429.206878] Lustre: server umount lustre-MDT0001 complete [ 1440.821230] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:07:20 (1783948040) [ 1447.612599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.117408] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.141545] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1448.238558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1452.801228] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1453.538524] LustreError: 50492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.659996] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.555085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.996201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1467.142820] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1475.682462] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1481.554811] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1488.658132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1489.074957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.745195] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1504.369352] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1510.355848] LustreError: 53018:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1511.344445] Lustre: Modifying parameter general.timeout=1 in log params [ 1513.179539] Lustre: Modifying parameter general.timeout=3 in log params [ 1513.183781] Lustre: Skipped 1 previous similar message [ 1515.869622] Lustre: Modifying parameter general.timeout=6 in log params [ 1515.872306] Lustre: Skipped 2 previous similar messages [ 1520.456572] Lustre: Modifying parameter general.timeout=11 in log params [ 1520.460732] Lustre: Skipped 4 previous similar messages [ 1528.799991] Lustre: Modifying parameter general.timeout=20 in log params [ 1528.806249] Lustre: Skipped 8 previous similar messages [ 1545.668153] Lustre: Modifying parameter general.timeout=38 in log params [ 1545.674636] Lustre: Skipped 17 previous similar messages [ 1577.693217] Lustre: Modifying parameter general.timeout=67 in log params [ 1577.697418] Lustre: Skipped 28 previous similar messages [ 1652.013986] Lustre: Disabling parameter general.timeout= in log params [ 1652.018591] Lustre: Skipped 62 previous similar messages [ 1655.653483] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:10:54 (1783948254) [ 1659.868873] Lustre: 59972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1661.586537] Lustre: 60070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1663.700461] Lustre: 60170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1668.477316] Lustre: 60385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1668.494789] Lustre: 60385:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1703.463848] Lustre: 61852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1703.481241] Lustre: 61852:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1703.516249] LustreError: 61852:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1708.115567] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:11:47 (1783948307) [ 1723.367712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1723.378030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.412850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1727.468346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.492329] Lustre: Skipped 1 previous similar message [ 1729.382507] Lustre: server umount lustre-MDT0000 complete [ 1732.585939] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1732.606671] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1733.649690] LustreError: 58872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948335 with bad export cookie 10017444530756240626 [ 1733.652884] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1733.676314] LustreError: 58872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1734.189566] Lustre: server umount lustre-MDT0001 complete [ 1748.943953] Lustre: server umount lustre-OST0000 complete [ 1779.410409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.818590] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1780.179734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.261343] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1794.667134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.782798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1794.836038] Lustre: 64384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1800.629894] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1808.989475] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1816.182928] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1824.409470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1824.618139] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1824.953221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.959041] Lustre: Skipped 1 previous similar message [ 1832.767636] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1841.592567] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1843.937981] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1843.956092] Lustre: Skipped 23 previous similar messages [ 1844.070189] Lustre: 66951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1856.995236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1857.000344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.047575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1857.051882] Lustre: Skipped 3 previous similar messages [ 1859.219614] Lustre: server umount lustre-OST0000 complete [ 1867.233382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1867.242225] LustreError: Skipped 1 previous similar message [ 1869.738875] Lustre: server umount lustre-MDT0000 complete [ 1872.866174] LustreError: 66253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.875678] LustreError: 66253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1874.282293] LustreError: 64378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948476 with bad export cookie 10017444530756243776 [ 1874.289989] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1874.293035] LustreError: 64378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1874.670483] Lustre: server umount lustre-MDT0001 complete [ 1887.418858] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1890.698763] Key type lgssc unregistered [ 1890.993779] LNet: 68162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.002314] LNetError: 68162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1891.034278] LNet: Removed LNI 192.168.203.109@tcp [ 1891.942337] Key type .llcrypt unregistered [ 1891.945084] Key type ._llcrypt unregistered [ 1919.806845] Key type ._llcrypt registered [ 1919.814665] Key type .llcrypt registered [ 1919.971377] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1921.277550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1921.474486] alg: No test for adler32 (adler32-zlib) [ 1922.689730] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 1923.096948] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1924.840240] Key type lgssc registered [ 1926.129773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1933.847200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1933.868650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.355552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.231531] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1945.574572] LustreError: 69579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.595731] LustreError: 69579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1950.345764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.049553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1957.747681] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1967.467129] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1974.835319] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1984.127172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1984.883592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.023259] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2004.116861] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2017.029545] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2031.075871] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.081126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.094392] Lustre: Skipped 1 previous similar message [ 2031.106291] Lustre: Skipped 2 previous similar messages [ 2035.178096] Lustre: server umount lustre-MDT0000 complete [ 2036.193705] LustreError: 69580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.226267] LustreError: 69580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2039.278642] LustreError: 69559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948641 with bad export cookie 1701997246896663823 [ 2039.294648] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.731875] Lustre: server umount lustre-MDT0001 complete [ 2054.588309] Lustre: server umount lustre-OST0000 complete [ 2085.944762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.302073] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2086.343649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2086.772784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.880341] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2101.426192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.639043] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2101.671340] Lustre: 74455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2102.490899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2108.043371] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2116.536991] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2123.001527] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2130.585399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.783508] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2131.436120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.784965] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2147.401118] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2149.986601] Lustre: 77033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2153.151338] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2164.197144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2164.213882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.216627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.224903] Lustre: Skipped 2 previous similar messages [ 2164.233175] Lustre: Skipped 1 previous similar message [ 2165.840208] Lustre: server umount lustre-OST0000 complete [ 2173.923561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2173.941185] LustreError: Skipped 1 previous similar message [ 2173.949766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.986137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.134535] Lustre: server umount lustre-MDT0000 complete [ 2179.553700] LustreError: 74462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.573237] LustreError: 74462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2181.847692] LustreError: 76337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948783 with bad export cookie 1701997246896664670 [ 2181.850574] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2181.859208] LustreError: 76337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2182.423442] Lustre: server umount lustre-MDT0001 complete [ 2194.534122] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2197.129047] Key type lgssc unregistered [ 2197.503896] LNet: 78246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2197.514394] LNetError: 78246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2197.529084] LNet: Removed LNI 192.168.203.109@tcp [ 2198.453495] Key type .llcrypt unregistered [ 2198.459887] Key type ._llcrypt unregistered [ 2226.512701] Key type ._llcrypt registered [ 2226.514327] Key type .llcrypt registered [ 2226.584247] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2227.277985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2227.329717] alg: No test for adler32 (adler32-zlib) [ 2228.554945] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 2229.063056] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2230.825111] Key type lgssc registered [ 2232.565639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2241.359372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2241.376285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.105144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.617822] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2252.258996] LustreError: 79666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.283569] LustreError: 79666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2257.382567] LustreError: 79667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.268715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.334354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2266.424297] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2277.377833] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2285.310399] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2292.654314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.178435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.793590] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2307.997700] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2318.626844] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2319.858378] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2321.731472] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2321.738523] Lustre: Skipped 1 previous similar message [ 2324.607618] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2324.611912] Lustre: Skipped 2 previous similar messages [ 2328.985525] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2328.989852] Lustre: Skipped 3 previous similar messages [ 2337.079272] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2337.092616] Lustre: Skipped 8 previous similar messages [ 2354.038388] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2354.042021] Lustre: Skipped 16 previous similar messages [ 2386.313963] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2386.318813] Lustre: Skipped 35 previous similar messages [ 2417.134726] Lustre: 87773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2419.737565] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:23:39 (1783949019) [ 2421.049869] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2422.797325] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:23:42 (1783949022) [ 2426.349144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.356863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2426.368365] Lustre: Skipped 1 previous similar message [ 2426.368835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2426.390836] Lustre: Skipped 1 previous similar message [ 2431.472350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2431.483872] Lustre: Skipped 1 previous similar message [ 2436.579423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2440.160184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2440.401365] Lustre: server umount lustre-OST0000 complete [ 2446.823111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.831915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2446.839234] Lustre: Skipped 1 previous similar message [ 2446.847939] Lustre: Skipped 2 previous similar messages [ 2450.687297] Lustre: server umount lustre-MDT0000 complete [ 2451.939309] LustreError: 83570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.954305] LustreError: 83570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2454.134259] LustreError: 84043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949055 with bad export cookie 17191181189283005269 [ 2454.136559] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.144022] LustreError: 84043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2454.483331] Lustre: server umount lustre-MDT0001 complete [ 2464.762092] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2467.495975] Key type lgssc unregistered [ 2467.750398] LNet: 89021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.760882] LNetError: 89021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.778685] LNet: Removed LNI 192.168.203.109@tcp [ 2468.575695] Key type .llcrypt unregistered [ 2468.579107] Key type ._llcrypt unregistered [ 2472.700636] Key type ._llcrypt registered [ 2472.703532] Key type .llcrypt registered [ 2472.827550] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2477.788937] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules [ 2479.245596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.307753] alg: No test for adler32 (adler32-zlib) [ 2480.399479] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 2480.697768] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2482.408230] Key type lgssc registered [ 2483.706105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.000547] LustreError: 90224:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2503.868469] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2507.280289] LustreError: 90224:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2516.641676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2516.673897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.968995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.746804] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2527.202430] LustreError: 91250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.229077] LustreError: 91250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2532.322543] LustreError: 91251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.991435] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:25:34 (1783949134) [ 2537.445910] LustreError: 91250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.506726] LustreError: 91283:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2539.535825] LustreError: 91283:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2539.555914] LustreError: 91283:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2539.949389] Lustre: server umount lustre-MDT0000 complete [ 2553.559766] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2556.434434] Key type lgssc unregistered [ 2556.778768] LNet: 92778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2556.800803] LNetError: 92778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2556.826313] LNet: Removed LNI 192.168.203.109@tcp [ 2557.581811] Key type .llcrypt unregistered [ 2557.584376] Key type ._llcrypt unregistered [ 2575.593516] Key type ._llcrypt registered [ 2575.597876] Key type .llcrypt registered [ 2575.699201] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2576.726902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2576.879960] alg: No test for adler32 (adler32-zlib) [ 2578.021775] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 2578.297520] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2580.008172] Key type lgssc registered [ 2581.128617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2588.734799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2588.762932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.268689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.688346] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2600.423514] LustreError: 94002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.441976] LustreError: 94002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2602.320822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.015480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2607.446957] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2615.651675] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2622.245676] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2628.536274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2629.087259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.649895] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2643.517176] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2694.567319] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:28:13 (1783949293) [ 2696.448509] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2698.246128] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:28:17 (1783949297) [ 2703.845097] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.854351] Lustre: Skipped 1 previous similar message [ 2703.857205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2703.879269] Lustre: Skipped 2 previous similar messages [ 2708.966627] LustreError: 94732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.979688] LustreError: 94732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2709.021292] Lustre: server umount lustre-MDT0000 complete [ 2712.579325] LustreError: 93982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949314 with bad export cookie 13365498823200319543 [ 2712.591124] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2712.594467] LustreError: 93982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2712.853863] Lustre: server umount lustre-MDT0001 complete [ 2727.047567] Lustre: server umount lustre-OST0000 complete [ 2738.695365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.184375] LustreError: 98277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.354212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.702813] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2744.801956] LustreError: 98278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.924439] LustreError: 98277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.959953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.742779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2756.826756] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2764.605857] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2770.686391] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2784.161234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2784.364167] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2784.372556] LustreError: 98267:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2784.417431] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2784.434537] LustreError: 100362:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2784.441397] LustreError: 100362:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2784.445877] LustreError: 100362:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2784.544543] Lustre: server umount lustre-OST0000 complete [ 2784.559542] LustreError: 100362:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2793.077960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2793.309570] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2793.315779] LustreError: 98267:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2793.331555] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2793.339755] LustreError: 100828:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2793.345566] LustreError: 100828:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2793.355246] LustreError: 100828:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2793.429695] Lustre: server umount lustre-OST0000 complete [ 2793.433225] LustreError: 100828:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2803.291337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2803.506245] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2803.510267] Lustre: Found index 0 for lustre-OST0000, updating log [ 2803.517840] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2803.603566] Lustre: lustre-OST0000: new disk, initializing [ 2803.607399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2803.616939] Lustre: 101362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2803.835882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.139481] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2815.886650] Lustre: Failing over lustre-OST0000 [ 2816.165824] Lustre: server umount lustre-OST0000 complete [ 2820.585657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.590067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2820.610187] Lustre: Skipped 2 previous similar messages [ 2820.627682] Lustre: Skipped 1 previous similar message [ 2822.691531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.746386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.750725] Lustre: Skipped 1 previous similar message [ 2830.899498] Lustre: server umount lustre-MDT0000 complete [ 2832.868197] LustreError: 99338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.887403] LustreError: 99338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2835.232676] LustreError: 98259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949437 with bad export cookie 13365498823200322287 [ 2835.239258] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2835.247692] LustreError: 98259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2836.083382] Lustre: server umount lustre-MDT0001 complete [ 2846.016467] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:30:45 (1783949445) [ 2864.249876] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2876.165261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.647576] LustreError: 103709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.752992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2881.562859] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2890.694705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2896.609956] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2899.726921] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2906.809176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.696097] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2914.040640] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2915.878600] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2915.882377] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2915.908930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2920.947933] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2920.995956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2922.023889] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2922.107123] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2922.116095] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2922.134313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2922.168764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2923.891187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.033835] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2924.291641] Lustre: lustre-OST0001: new disk, initializing [ 2924.305584] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2924.323128] Lustre: 106032:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2924.859183] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2924.876846] Lustre: Skipped 2 previous similar messages [ 2929.913105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2929.920613] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2930.036880] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2932.659534] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2941.965105] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2949.556716] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2955.758917] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.782111] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2955.789739] Lustre: Skipped 2 previous similar messages [ 2965.991991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2966.003762] Lustre: Skipped 7 previous similar messages [ 2968.032124] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2968.282832] Lustre: server umount lustre-MDT0001 complete [ 2971.136753] LustreError: 103709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.173275] LustreError: 103709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2977.643488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.840617] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2977.851060] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2977.924600] Lustre: 103699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2979.189907] LustreError: 107559:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2979.203477] LustreError: 107559:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2979.236520] LustreError: 103701:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2979.428344] LustreError: 107572:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.109@tcp: cfg command failed: rc = -17 [ 2979.441835] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2979.441835] [ 2983.940507] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2985.039448] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2994.819716] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:33:14 (1783949594) [ 3010.849751] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3028.563500] Lustre: 109648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3050.339421] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3054.393191] Lustre: 110783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3142.112919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3142.125469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.149127] Lustre: Skipped 2 previous similar messages [ 3142.153150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3142.156256] Lustre: Skipped 1 previous similar message [ 3145.937650] Lustre: server umount lustre-MDT0000 complete [ 3147.745629] LustreError: 103706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.767350] LustreError: 103706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3149.333298] LustreError: 106031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949751 with bad export cookie 13365498823200323288 [ 3149.336611] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3149.345075] LustreError: 106031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3149.637130] Lustre: server umount lustre-MDT0001 complete [ 3163.704632] Lustre: server umount lustre-OST0000 complete [ 3178.315178] Lustre: server umount lustre-OST0001 complete [ 3183.399304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3193.076561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3201.527956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3211.422254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3229.290799] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3239.795676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.831453] Lustre: lustre-MDT0000: reset Object Index mappings [ 3240.438219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.443705] Lustre: Skipped 1 previous similar message [ 3245.277861] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3253.764925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.784989] Lustre: lustre-MDT0001: reset Object Index mappings [ 3259.610575] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3262.471264] Lustre: 115882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3268.234390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3274.468762] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3279.092286] LustreError: 116234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.112967] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3279.113781] LustreError: 116234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3282.734211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.367367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3285.374297] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3285.381508] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3289.313873] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3296.517919] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3299.627180] Lustre: 117758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3356.035458] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:39:15 (1783949955) [ 3382.758836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.771271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.777373] Lustre: Skipped 4 previous similar messages [ 3382.792940] Lustre: Skipped 5 previous similar messages [ 3387.365038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3388.800701] Lustre: server umount lustre-MDT0000 complete [ 3392.591295] LustreError: 114714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949994 with bad export cookie 13365498823200370363 [ 3392.606542] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3392.606637] LustreError: 114714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3392.997373] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.011734] Lustre: Skipped 2 previous similar messages [ 3399.118908] Lustre: server umount lustre-MDT0001 complete [ 3409.250643] Lustre: server umount lustre-OST0000 complete [ 3419.997712] Lustre: server umount lustre-OST0001 complete [ 3427.807759] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3439.484651] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3478.054341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.274547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3478.309816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3478.398258] Lustre: lustre-MDT0000: new disk, initializing [ 3478.509120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.517586] Lustre: Skipped 3 previous similar messages [ 3478.543969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3483.548101] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3490.923558] Lustre: server umount lustre-MDT0000 complete [ 3500.389677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.715657] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3500.730441] Lustre: 122025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3500.753376] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3504.746685] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3509.733591] Lustre: server umount lustre-MDT0000 complete [ 3519.620399] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:41:58 (1783950118) [ 3521.452272] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3523.674575] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:42:02 (1783950122) [ 3543.466678] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3554.268110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.770857] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3569.428539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.555037] Lustre: 124076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3569.669357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3569.673260] Lustre: Skipped 1 previous similar message [ 3569.756346] Lustre: lustre-MDT0001: new disk, initializing [ 3569.909812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3569.927674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3575.181585] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3580.342749] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3587.620309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3587.903633] Lustre: lustre-OST0000: new disk, initializing [ 3587.907562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3587.913997] Lustre: 125656:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3589.881431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3589.889873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3590.005452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3594.452725] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3605.576630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3605.799876] Lustre: lustre-OST0001: new disk, initializing [ 3605.803907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3605.817128] Lustre: 126526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3607.551491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3607.558641] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3607.638779] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3612.817475] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3623.518225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3627.380588] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4014.126690] Lustre: 132941:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4014.134139] Lustre: 132941:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4014.146518] Lustre: 132941:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4014.166743] Lustre: 132941:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4014.177606] Lustre: 132941:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4014.187667] Lustre: 132941:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4329.594515] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:55:28 (1783950928) [ 4347.884594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.888812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4347.901675] Lustre: Skipped 4 previous similar messages [ 4347.910390] Lustre: Skipped 11 previous similar messages [ 4351.055887] Lustre: server umount lustre-MDT0000 complete [ 4352.999849] LustreError: 124083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.008689] LustreError: 124083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4355.147281] LustreError: 126525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950956 with bad export cookie 13365498823200440006 [ 4355.155492] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4355.161132] LustreError: 126525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4358.124616] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.131897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4358.138940] Lustre: Skipped 1 previous similar message [ 4358.158760] Lustre: Skipped 1 previous similar message [ 4361.635212] Lustre: server umount lustre-MDT0001 complete [ 4371.238934] Lustre: server umount lustre-OST0000 complete [ 4380.654768] Lustre: server umount lustre-OST0001 complete [ 4387.974960] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 4398.381475] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4436.516406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.740097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4436.762530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4436.828786] Lustre: lustre-MDT0000: new disk, initializing [ 4436.892214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.897369] Lustre: Skipped 5 previous similar messages [ 4436.909781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4441.012838] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4454.829256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4454.921465] Lustre: 136000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4455.010170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4455.018494] Lustre: Skipped 1 previous similar message [ 4455.140744] Lustre: lustre-MDT0001: new disk, initializing [ 4455.319315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4455.339022] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4461.093888] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4471.283559] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4478.238863] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4486.049997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4486.477737] Lustre: lustre-OST0000: new disk, initializing [ 4486.487283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4486.504776] Lustre: 137968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4486.674890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4486.686484] Lustre: Skipped 1 previous similar message [ 4488.109827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4488.125332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4488.273466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4493.233754] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4503.827692] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4513.873937] Lustre: lustre-MDD0000: changelog on [ 4516.841913] Lustre: lustre-MDD0001: changelog on [ 6658.896827] Lustre: lustre-MDD0001: changelog off [ 6660.353414] Lustre: 137074:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6661.181093] Lustre: lustre-MDD0000: changelog off [ 6669.285808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6669.287400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.289912] LustreError: Skipped 1 previous similar message [ 6669.301373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6669.315775] Lustre: Skipped 3 previous similar messages [ 6672.596514] Lustre: server umount lustre-OST0000 complete [ 6677.473103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6677.483394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6677.509659] Lustre: Skipped 1 previous similar message [ 6677.520447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6677.527843] Lustre: Skipped 2 previous similar messages [ 6680.545978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6682.671888] Lustre: server umount lustre-MDT0000 complete [ 6685.669761] LustreError: 136007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.695117] LustreError: 136007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6686.109461] LustreError: 135990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953287 with bad export cookie 13365498823201262191 [ 6686.114986] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6686.117581] LustreError: 135990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6686.449546] Lustre: server umount lustre-MDT0001 complete [ 6695.750354] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 6698.452629] Key type lgssc unregistered [ 6698.725834] LNet: 147148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6698.741719] LNetError: 147148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6698.772117] LNet: Removed LNI 192.168.203.109@tcp [ 6699.621219] Key type .llcrypt unregistered [ 6699.626988] Key type ._llcrypt unregistered [ 6722.609698] Key type ._llcrypt registered [ 6722.612168] Key type .llcrypt registered [ 6722.850426] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 6742.732539] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6744.597721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6744.621777] alg: No test for adler32 (adler32-zlib) [ 6745.842932] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 6746.257103] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 6747.976438] Key type lgssc registered [ 6749.350751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6790.020915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6790.081846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.352532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6791.389446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6791.503897] Lustre: lustre-MDT0000: new disk, initializing [ 6791.604215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6791.627287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6796.254755] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6805.649099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6805.768065] Lustre: 150353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6805.823932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6805.827752] Lustre: Skipped 1 previous similar message [ 6805.885958] Lustre: lustre-MDT0001: new disk, initializing [ 6805.979512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6806.030963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6806.050734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6810.467512] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6820.798157] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6826.505302] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6832.943835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6833.185825] Lustre: lustre-OST0000: new disk, initializing [ 6833.189574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6833.196464] Lustre: 152322:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6833.286036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6839.867374] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6841.439625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6841.448662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6841.562171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6849.470254] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6854.431539] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_module obdecho/obdecho [ 6856.200311] LustreError: 153325:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6856.209678] LustreError: 153325:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6857.839563] LustreError: 153422:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6872.549543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6872.556556] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6872.561626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6872.581293] Lustre: Skipped 2 previous similar messages [ 6874.807464] Lustre: server umount lustre-MDT0000 complete [ 6878.465318] LustreError: 151147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953480 with bad export cookie 4289205811885803005 [ 6878.469063] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6878.476480] LustreError: 151147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6878.879619] Lustre: server umount lustre-MDT0001 complete [ 6892.772542] Lustre: server umount lustre-OST0000 complete [ 6916.719332] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 6928.173213] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6965.472049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6965.739088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6965.771419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6965.833932] Lustre: lustre-MDT0000: new disk, initializing [ 6965.895412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6965.910707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6971.053478] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6983.149269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6983.286172] Lustre: 157137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6983.343771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6983.356672] Lustre: Skipped 1 previous similar message [ 6983.427319] Lustre: lustre-MDT0001: new disk, initializing [ 6983.524664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6983.593813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6983.604334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6989.251578] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6998.809476] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7004.802095] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7012.697658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7013.201679] Lustre: lustre-OST0000: new disk, initializing [ 7013.215660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7013.248422] Lustre: 159107:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7013.400965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7018.619812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7018.624500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7018.815397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7022.190387] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7034.037064] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7039.095219] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7039.273989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7044.149151] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7044.373510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7049.699225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7049.715645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7049.736576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7049.745584] Lustre: Skipped 3 previous similar messages [ 7052.934089] Lustre: server umount lustre-OST0000 complete [ 7059.939146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7059.940046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7059.945361] Lustre: Skipped 2 previous similar messages [ 7059.951610] Lustre: Skipped 1 previous similar message [ 7063.188686] Lustre: server umount lustre-MDT0000 complete [ 7065.065564] LustreError: 157149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7065.091774] LustreError: 157149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7067.537196] LustreError: 157130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953669 with bad export cookie 4289205811885803964 [ 7067.555989] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7068.105716] Lustre: server umount lustre-MDT0001 complete [ 7072.275704] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:41:11 (1783953671) [ 7074.143097] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7076.233431] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:41:15 (1783953675) [ 7084.296313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7084.686300] LustreError: 161519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.773043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7089.531087] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7090.150437] LustreError: 161520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.251583] LustreError: 161519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7098.959207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.013736] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7113.726282] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7121.602700] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7130.204107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7130.598425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7130.605897] Lustre: Skipped 1 previous similar message [ 7137.961325] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7145.834768] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7165.414124] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7165.429184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7165.456107] Lustre: Skipped 2 previous similar messages [ 7165.462358] Lustre: Skipped 2 previous similar messages [ 7168.358904] Lustre: server umount lustre-MDT0000 complete [ 7170.534790] LustreError: 161515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7170.552825] LustreError: 161515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7172.525175] LustreError: 162255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953774 with bad export cookie 4289205811885804846 [ 7172.527164] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7172.540594] LustreError: 162255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7175.650526] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7175.671269] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7175.682965] Lustre: Skipped 1 previous similar message [ 7179.009441] Lustre: server umount lustre-MDT0001 complete [ 7185.855099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7199.862416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.032992] LustreError: 165387:0:(llog.c:1655:llog_backup()) MGC192.168.203.109@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7200.041674] Lustre: 165387:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.109@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7202.273276] LustreError: 165392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7202.305434] LustreError: 165392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7218.662450] LustreError: 165393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.689842] LustreError: 165393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7253.898536] LustreError: 165394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7253.938087] LustreError: 165394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 7260.640664] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7260.654162] Lustre: 165387:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.109@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7301.639518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7306.098819] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7306.727756] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7314.461228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.407352] LustreError: 165393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7318.439816] LustreError: 165393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 7319.014460] LustreError: 148574:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000b8ccff1 x1870610665965568/t0(0) o250->MGC192.168.203.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7324.674359] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7324.982985] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7333.474969] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7335.307057] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7341.769518] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7343.563072] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7365.603438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7365.606029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7365.606426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7365.606430] LustreError: Skipped 1 previous similar message [ 7365.628060] Lustre: Skipped 2 previous similar messages [ 7365.655858] Lustre: Skipped 4 previous similar messages [ 7366.490850] Lustre: server umount lustre-MDT0000 complete [ 7370.336130] LustreError: 161501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953972 with bad export cookie 4289205811885811300 [ 7370.339327] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7370.354849] LustreError: 161501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7370.891805] Lustre: server umount lustre-MDT0001 complete [ 7377.440272] Lustre: server umount lustre-OST0000 complete [ 7398.615414] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 7412.986794] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7454.526252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.973409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7455.063523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7455.179981] Lustre: lustre-MDT0000: new disk, initializing [ 7455.333793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.348186] Lustre: Skipped 1 previous similar message [ 7455.387696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7460.257743] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7471.964562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7472.115431] Lustre: 170516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7472.410144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7477.654760] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7488.277651] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7495.278144] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7502.933185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7503.290996] Lustre: lustre-OST0000: new disk, initializing [ 7503.293482] Lustre: Skipped 1 previous similar message [ 7503.298899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7503.319582] Lustre: Skipped 2 previous similar messages [ 7503.334798] Lustre: 172489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7510.785584] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7511.134624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7511.150847] Lustre: Skipped 1 previous similar message [ 7511.156587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7511.217893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7520.947108] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7526.169509] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7526.422204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7532.125545] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7532.355895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7537.124512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7537.130970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7537.139325] Lustre: Skipped 1 previous similar message [ 7537.144258] LustreError: Skipped 1 previous similar message [ 7537.147024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7537.156091] Lustre: Skipped 1 previous similar message [ 7540.622412] Lustre: server umount lustre-OST0000 complete [ 7550.846613] Lustre: server umount lustre-MDT0000 complete [ 7552.483158] LustreError: 171743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7552.507902] LustreError: 171743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7554.971179] LustreError: 170510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954156 with bad export cookie 4289205811885828135 [ 7554.981369] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7554.990027] LustreError: 170510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7555.315248] Lustre: server umount lustre-MDT0001 complete [ 7564.979659] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:49:24 (1783954164) [ 7571.925651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.874196] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7588.735321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.229696] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7605.555879] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7611.225692] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7618.693665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7626.053988] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7634.431877] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7655.923573] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:50:55 (1783954255) [ 7660.518573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7660.521354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7660.538894] Lustre: Skipped 2 previous similar messages [ 7660.548612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7660.553967] Lustre: Skipped 2 previous similar messages [ 7666.425431] Lustre: server umount lustre-OST0000 complete [ 7671.268165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7671.278479] LustreError: Skipped 1 previous similar message [ 7676.573236] Lustre: server umount lustre-MDT0000 complete [ 7679.901146] LustreError: 174883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954281 with bad export cookie 4289205811885829017 [ 7679.908966] LustreError: 174883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7679.912773] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7680.228160] Lustre: server umount lustre-MDT0001 complete [ 7691.634776] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 7694.395793] Key type lgssc unregistered [ 7694.818429] LNet: 178652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7694.828106] LNetError: 178652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7694.842422] LNet: Removed LNI 192.168.203.109@tcp [ 7695.904332] Key type .llcrypt unregistered [ 7695.910168] Key type ._llcrypt unregistered [ 7717.292188] Key type ._llcrypt registered [ 7717.295784] Key type .llcrypt registered [ 7717.414366] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7718.564877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7718.633626] alg: No test for adler32 (adler32-zlib) [ 7719.763424] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 7719.965440] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 7721.672490] Key type lgssc registered [ 7722.757784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7730.842928] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7730.870755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7746.464474] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7751.587134] LustreError: 179895:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 7756.769119] LustreError: 179895:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7756.779779] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7756.784784] LustreError: MGC192.168.203.109@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7756.795602] LustreError: 179895:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7756.813546] LustreError: 179895:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7756.826679] LustreError: 179895:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7756.944973] Lustre: server umount lustre-OST0000 complete [ 7756.949812] LustreError: 179895:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7766.421898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7767.883058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7772.548248] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7777.254495] LustreError: 180395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7777.291078] LustreError: 180395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7781.084868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7781.496103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7786.950585] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7796.265253] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7803.746253] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7811.580247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7811.867566] LustreError: 182212:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7811.900859] Lustre: 182212:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.109@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7812.094179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.518489] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7827.366129] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7831.524810] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7831.531295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7831.539546] Lustre: Skipped 2 previous similar messages [ 7831.554749] Lustre: Skipped 2 previous similar messages [ 7835.780221] Lustre: server umount lustre-MDT0000 complete [ 7836.650405] LustreError: 181124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7836.681706] LustreError: 181124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7839.650145] LustreError: 180375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954441 with bad export cookie 11631947652887360720 [ 7839.660514] LustreError: 180375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7839.668852] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7840.038559] Lustre: server umount lustre-MDT0001 complete [ 7854.731230] Lustre: server umount lustre-OST0000 complete [ 7865.035254] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 7876.698973] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7917.398442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.769623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7917.810125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7917.891307] Lustre: lustre-MDT0000: new disk, initializing [ 7917.955406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7917.987710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7922.619113] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7932.777305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.930171] Lustre: 185741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7933.020683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7933.028899] Lustre: Skipped 1 previous similar message [ 7933.138903] Lustre: lustre-MDT0001: new disk, initializing [ 7933.234945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7933.260074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7933.272523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7938.464303] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7949.067609] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7955.652567] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7962.395886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7962.707303] Lustre: lustre-OST0000: new disk, initializing [ 7962.711239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7962.719270] Lustre: 187707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7962.796084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7969.081762] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7969.357817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7969.369142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7969.463450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7978.211382] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7983.178378] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7983.382291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7988.560949] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7988.868132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7995.362111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7995.370207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7995.382341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8000.514295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8000.526252] Lustre: Skipped 2 previous similar messages [ 8005.088240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8005.365486] Lustre: server umount lustre-OST0000 complete [ 8010.211912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8010.222739] LustreError: Skipped 1 previous similar message [ 8010.229115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8010.244500] Lustre: Skipped 1 previous similar message [ 8010.250526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8015.852122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8015.860052] Lustre: Skipped 1 previous similar message [ 8016.317837] Lustre: server umount lustre-MDT0000 complete [ 8020.477468] LustreError: 185732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954622 with bad export cookie 11631947652887361441 [ 8020.500929] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8020.505385] LustreError: 185732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8021.094916] Lustre: server umount lustre-MDT0001 complete [ 8032.838768] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:57:11 (1783954631) [ 8050.917901] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 8053.461924] Key type lgssc unregistered [ 8053.720614] LNet: 190655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8053.726231] LNetError: 190655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8054.763508] LNet: Removed LNI 192.168.203.109@tcp [ 8055.632336] Key type .llcrypt unregistered [ 8055.634220] Key type ._llcrypt unregistered [ 8073.821565] Key type ._llcrypt registered [ 8073.824423] Key type .llcrypt registered [ 8073.960784] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8075.160326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8075.374773] alg: No test for adler32 (adler32-zlib) [ 8076.587214] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 8076.833695] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 8078.553790] Key type lgssc registered [ 8079.736767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8088.042482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8088.052902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8089.424776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8094.520604] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8099.813628] LustreError: 191875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8099.842872] LustreError: 191875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8103.406334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8103.743361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8108.851162] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8116.295845] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8121.885712] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8130.008802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8130.193403] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8130.197117] LustreError: 193742:0:(llog.c:1696:llog_backup()) MGC192.168.203.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8130.204493] Lustre: 193742:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.109@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8130.327881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8136.487757] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8144.576589] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8149.794485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8149.804240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8149.820501] Lustre: Skipped 1 previous similar message [ 8149.844894] Lustre: Skipped 2 previous similar messages [ 8153.088992] Lustre: server umount lustre-MDT0000 complete [ 8155.124420] LustreError: 192867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8155.155511] LustreError: 192867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8156.882952] LustreError: 191855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954758 with bad export cookie 10661278376399389666 [ 8156.890295] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8156.895482] LustreError: 191855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8158.113947] LustreError: 192605:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000030774178 x1870612060466560/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8158.289290] Lustre: server umount lustre-MDT0001 complete [ 8172.977769] Lustre: server umount lustre-OST0000 complete [ 8183.171256] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 8194.626629] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8236.631437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8236.902756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8236.948533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8237.084615] Lustre: lustre-MDT0000: new disk, initializing [ 8237.204601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8237.226046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8242.934447] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8255.460550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8255.658668] Lustre: 197266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8255.730060] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8255.732614] Lustre: Skipped 1 previous similar message [ 8255.736644] LustreError: 198050:0:(llog.c:1696:llog_backup()) MGC192.168.203.109@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8255.742826] LustreError: 198050:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8255.759378] Lustre: 198050:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.109@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8255.787730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8255.804688] Lustre: Skipped 1 previous similar message [ 8255.923626] Lustre: lustre-MDT0001: new disk, initializing [ 8256.037243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8256.095357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8256.115385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8261.665089] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8272.294745] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8278.525157] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8286.106543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.429875] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8286.434995] Lustre: Skipped 1 previous similar message [ 8286.440845] LustreError: 199224:0:(llog.c:1696:llog_backup()) MGC192.168.203.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8286.453642] LustreError: 199224:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8286.461966] Lustre: 199224:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.109@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8286.470692] Lustre: 199224:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8286.500850] Lustre: lustre-OST0000: new disk, initializing [ 8286.516778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8286.524076] Lustre: 199247:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8286.649136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8292.971799] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8295.453487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8295.461224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8295.610924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8302.700714] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8307.943233] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8308.133687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8313.117707] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8313.373137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8315.873994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8315.891499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.906665] Lustre: Skipped 1 previous similar message [ 8315.919651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8316.393600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8321.484572] Lustre: server umount lustre-OST0000 complete [ 8326.626793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8326.635324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8326.642956] Lustre: Skipped 2 previous similar messages [ 8326.663224] Lustre: Skipped 3 previous similar messages [ 8331.328541] Lustre: server umount lustre-MDT0000 complete [ 8331.747258] LustreError: 197279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8331.766018] LustreError: 197279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8334.697098] LustreError: 197259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954936 with bad export cookie 10661278376399390387 [ 8334.705399] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8334.715425] LustreError: 197259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8334.981129] Lustre: server umount lustre-MDT0001 complete [ 8343.316712] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:02:22 (1783954942) [ 8360.684810] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8371.462184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8371.908314] LustreError: 202375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.984780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.319787] LustreError: 202376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8377.390800] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8386.371897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8386.534658] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8391.572785] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8394.844766] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8401.384841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8408.872893] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8417.995310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8418.176306] Lustre: lustre-OST0001: new disk, initializing [ 8418.179712] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8418.186977] Lustre: 204689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8418.304219] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8418.317021] Lustre: Skipped 2 previous similar messages [ 8423.532942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8423.550656] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8423.618938] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8425.441923] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8437.082920] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8445.424112] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8447.771276] Lustre: DEBUG MARKER: ADD OST3 [ 8453.916037] loop: module loaded [ 8454.028604] loop0: detected capacity change from 0 to 204800000 [ 8454.263792] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8454.292161] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8454.318710] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8461.150049] loop0: detected capacity change from 0 to 204800000 [ 8461.484651] Lustre: DEBUG MARKER: STOP OST3 [ 8467.528643] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8467.812909] Lustre: lustre-OST0003: new disk, initializing [ 8467.818647] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8467.825317] Lustre: 206725:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8469.294403] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8469.449240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8469.467828] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8469.505808] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8471.329082] LustreError: 206853:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8471.652110] Lustre: Failing over lustre-OST0003 [ 8471.750599] Lustre: server umount lustre-OST0003 complete [ 8474.557673] LustreError: 203863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8474.575068] LustreError: 203863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8474.603865] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8474.607762] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8475.658271] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8483.930880] Lustre: DEBUG MARKER: START OST3 again [ 8491.836781] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8492.300986] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8492.312875] Lustre: Skipped 1 previous similar message [ 8492.330411] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8493.803242] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8494.085586] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8494.085896] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8498.990654] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8519.766054] Lustre: Failing over lustre-OST0003 [ 8519.859953] Lustre: server umount lustre-OST0003 complete [ 8523.262720] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8523.281435] Lustre: Skipped 1 previous similar message [ 8523.292085] LustreError: 203863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8523.316137] LustreError: 203863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8540.643944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8540.653391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8540.664785] Lustre: Skipped 1 previous similar message [ 8540.670989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8545.374461] Lustre: server umount lustre-MDT0000 complete [ 8549.232247] LustreError: 202357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955151 with bad export cookie 10661278376399391269 [ 8549.234178] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8549.249377] LustreError: 202357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8553.957613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8553.960235] Lustre: Skipped 4 previous similar messages [ 8555.568139] Lustre: server umount lustre-MDT0001 complete [ 8561.201851] Lustre: server umount lustre-OST0000 complete [ 8566.560809] Lustre: server umount lustre-OST0001 complete [ 8578.041973] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 8587.301526] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8630.850284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.142180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8631.174627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8631.270951] Lustre: lustre-MDT0000: new disk, initializing [ 8631.457595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8631.499050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8636.309106] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8647.955149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8648.131706] Lustre: 212149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8648.184638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8648.194440] Lustre: Skipped 1 previous similar message [ 8648.274169] Lustre: lustre-MDT0001: new disk, initializing [ 8648.426490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8648.441408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8653.099072] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8662.449641] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8668.770841] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8675.521016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8675.819944] Lustre: 214117:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8676.986453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8677.146982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8682.172836] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8691.579535] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8696.232815] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8696.457845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8700.915608] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8701.169584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8708.066692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8708.078580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8708.084477] LustreError: Skipped 1 previous similar message [ 8708.085605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8708.085612] Lustre: Skipped 3 previous similar messages [ 8708.124322] Lustre: Skipped 6 previous similar messages [ 8709.454996] Lustre: server umount lustre-OST0000 complete [ 8718.996786] Lustre: server umount lustre-MDT0000 complete [ 8722.832598] LustreError: 212140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955324 with bad export cookie 10661278376399393558 [ 8722.835797] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8722.845561] LustreError: 212140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8723.211692] Lustre: server umount lustre-MDT0001 complete [ 8727.324093] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:08:46 (1783955326) [ 8744.868161] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 8755.810659] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8795.368356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8795.672665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8795.726454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8795.731883] Lustre: Skipped 1 previous similar message [ 8795.840838] Lustre: lustre-MDT0000: new disk, initializing [ 8795.845085] Lustre: Skipped 1 previous similar message [ 8796.013281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8796.030079] Lustre: Skipped 1 previous similar message [ 8801.288304] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8813.050098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8813.236405] Lustre: 218728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8813.411894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8817.933611] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8829.377677] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8836.632260] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8844.418072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8844.691759] Lustre: 220695:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8846.031048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8846.152755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8852.008177] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8861.631931] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8866.787494] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8867.035646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8871.265521] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8871.467870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8877.035937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.039053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8877.053230] Lustre: Skipped 2 previous similar messages [ 8877.054571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8877.073936] Lustre: Skipped 6 previous similar messages [ 8880.028683] Lustre: server umount lustre-OST0000 complete [ 8890.447132] Lustre: server umount lustre-MDT0000 complete [ 8892.386291] LustreError: 218738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.414917] LustreError: 218738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8894.265394] LustreError: 219530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955496 with bad export cookie 10661278376399394440 [ 8894.272393] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8894.281058] LustreError: 219530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8894.727422] Lustre: server umount lustre-MDT0001 complete [ 8903.847414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8904.420805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8904.428474] Lustre: Skipped 5 previous similar messages [ 8909.246279] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8919.277191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8925.347399] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8934.047719] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8939.452843] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8946.636689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8953.268987] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8961.799608] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8974.817484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8974.826500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8974.828646] LustreError: Skipped 2 previous similar messages [ 8974.830590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8974.840368] Lustre: Skipped 4 previous similar messages [ 8974.860450] Lustre: Skipped 4 previous similar messages [ 8978.563729] Lustre: server umount lustre-OST0000 complete [ 8997.344568] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8997.570460] Lustre: server umount lustre-MDT0000 complete [ 8999.393271] LustreError: 223884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.423269] LustreError: 223884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9001.786847] LustreError: 222894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955603 with bad export cookie 10661278376399395329 [ 9001.788170] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9001.799823] LustreError: 222894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9002.451867] Lustre: server umount lustre-MDT0001 complete [ 9103.479841] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 9105.900571] Key type lgssc unregistered [ 9106.222406] LNet: 226581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9106.238305] LNetError: 226581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9106.265210] LNet: Removed LNI 192.168.203.109@tcp [ 9107.121209] Key type .llcrypt unregistered [ 9107.124821] Key type ._llcrypt unregistered [ 9130.122966] Key type ._llcrypt registered [ 9130.124753] Key type .llcrypt registered [ 9130.226439] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 9143.822171] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9144.815802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9144.833456] alg: No test for adler32 (adler32-zlib) [ 9145.882343] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 9146.106840] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 9147.760240] Key type lgssc registered [ 9148.709620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9181.481044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9181.496377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9182.733540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9182.773429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9182.886363] Lustre: lustre-MDT0000: new disk, initializing [ 9182.978179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9183.003298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9186.944166] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9197.277141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.446036] Lustre: 229935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9197.502483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9197.504784] Lustre: Skipped 1 previous similar message [ 9197.603300] Lustre: lustre-MDT0001: new disk, initializing [ 9197.705601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9197.755304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9197.765883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9202.174335] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9211.882395] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9217.559674] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9224.688549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9225.014524] Lustre: lustre-OST0000: new disk, initializing [ 9225.021145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9225.031929] Lustre: 231901:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9225.153490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9231.306266] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9234.532564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9234.545530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9234.664210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9240.777413] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9245.151057] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9245.355297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9249.680146] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9249.832288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9255.394118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9255.396265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9255.405338] LustreError: Skipped 1 previous similar message [ 9255.406770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9255.425896] Lustre: Skipped 1 previous similar message [ 9255.439806] Lustre: Skipped 1 previous similar message [ 9257.774044] Lustre: server umount lustre-OST0000 complete [ 9264.610072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9264.632746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9264.667744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9265.644647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9268.190735] Lustre: server umount lustre-MDT0000 complete [ 9272.010651] LustreError: 229928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955873 with bad export cookie 8974311458851485922 [ 9272.019390] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9272.587693] Lustre: server umount lustre-MDT0001 complete [ 9292.087914] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9304.032284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9304.410292] LustreError: 234835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9304.434618] LustreError: 234835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9304.463526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9309.432335] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9309.678230] LustreError: 234836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9314.787817] LustreError: 234835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9318.076759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.457217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9322.976522] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9326.295759] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9333.910733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9334.556329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9342.115981] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9350.921786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9351.102473] Lustre: lustre-OST0001: new disk, initializing [ 9351.110078] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9351.119754] Lustre: 237141:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9351.237497] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9357.293582] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9360.028624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9360.038911] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9360.143685] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9369.000158] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9379.768633] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9392.128465] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:19:51 (1783955991) [ 9395.174552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9395.188525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9395.209586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9395.217180] Lustre: Skipped 2 previous similar messages [ 9400.608430] Lustre: server umount lustre-MDT0000 complete [ 9401.330958] LustreError: 237670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.361160] LustreError: 237670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9404.288321] LustreError: 234817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956006 with bad export cookie 8974311458851486804 [ 9404.294889] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9404.304411] LustreError: 234817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9404.696483] Lustre: server umount lustre-MDT0001 complete [ 9418.924827] Lustre: server umount lustre-OST0000 complete [ 9432.968136] Lustre: server umount lustre-OST0001 complete [ 9439.848694] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 9449.816667] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9485.850583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9486.051347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9486.073649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9486.145390] Lustre: lustre-MDT0000: new disk, initializing [ 9486.217900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9486.235983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9490.498558] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9500.761785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9500.863984] Lustre: 241091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9500.926626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9500.938693] Lustre: Skipped 1 previous similar message [ 9501.044789] Lustre: lustre-MDT0001: new disk, initializing [ 9501.236420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9501.244403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9505.109033] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9515.350351] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9521.988485] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9529.377148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9529.836443] Lustre: lustre-OST0000: new disk, initializing [ 9529.849214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9529.872248] Lustre: 243059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9531.434307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9531.440974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9531.647524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9536.757579] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9547.289979] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9552.016215] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9552.153301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9557.132143] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9557.493865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9562.597364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9562.608161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9562.611447] LustreError: Skipped 1 previous similar message [ 9562.613596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9562.623383] Lustre: Skipped 4 previous similar messages [ 9562.636360] Lustre: Skipped 5 previous similar messages [ 9565.717512] Lustre: server umount lustre-OST0000 complete [ 9572.835105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9572.849811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.856673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9572.881756] Lustre: Skipped 1 previous similar message [ 9576.896182] Lustre: server umount lustre-MDT0000 complete [ 9577.954632] LustreError: 241102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9577.993176] LustreError: 241102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9581.839973] LustreError: 241083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956183 with bad export cookie 8974311458851488939 [ 9581.843255] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9581.853265] LustreError: 241083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9582.200874] Lustre: server umount lustre-MDT0001 complete [ 9590.238936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9590.756460] LustreError: 245274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9590.828578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9590.842196] Lustre: Skipped 2 previous similar messages [ 9595.670846] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9604.615800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9609.901612] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9616.660632] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9621.870427] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9628.025804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9634.543515] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9641.076222] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9788.386140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9788.394950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.409358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9788.415086] Lustre: Skipped 1 previous similar message [ 9799.136210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9799.301811] Lustre: server umount lustre-OST0000 complete [ 9804.774938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9804.779817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9804.784280] LustreError: Skipped 1 previous similar message [ 9804.792879] Lustre: Skipped 5 previous similar messages [ 9808.994690] Lustre: server umount lustre-MDT0000 complete [ 9809.892420] LustreError: 246003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.918108] LustreError: 246003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9812.485179] LustreError: 245254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956414 with bad export cookie 8974311458851489821 [ 9812.487957] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9812.498391] LustreError: 245254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9812.870971] Lustre: server umount lustre-MDT0001 complete [ 9908.352766] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 9910.657539] Key type lgssc unregistered [ 9910.887735] LNet: 248882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9910.903539] LNetError: 248882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9910.915890] LNet: Removed LNI 192.168.203.109@tcp [ 9911.681314] Key type .llcrypt unregistered [ 9911.690078] Key type ._llcrypt unregistered [ 9936.258495] Key type ._llcrypt registered [ 9936.260180] Key type .llcrypt registered [ 9936.404189] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 9951.140263] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9951.887598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9951.897448] alg: No test for adler32 (adler32-zlib) [ 9953.069076] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 9953.280319] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 9954.904169] Key type lgssc registered [ 9955.970344] Lustre: Echo OBD driver; http://www.lustre.org/ [10002.515218] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10011.852201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10011.880919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10013.198923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10013.233893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10013.299282] Lustre: lustre-MDT0000: new disk, initializing [10013.359246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10013.379097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10017.987333] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10028.400638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.521923] Lustre: 252953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10028.562986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10028.567701] Lustre: Skipped 1 previous similar message [10028.627418] Lustre: lustre-MDT0001: new disk, initializing [10028.736020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10028.779129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10028.788666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10034.127975] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10038.873965] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10045.434769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10045.775624] Lustre: lustre-OST0000: new disk, initializing [10045.779764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10045.787809] Lustre: 254584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10045.889251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10051.911703] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10052.683433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10052.698456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10052.831824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10062.246676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10062.404974] Lustre: lustre-OST0001: new disk, initializing [10062.410850] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10062.427885] Lustre: 255457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10062.563775] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10068.044713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10068.052960] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10068.120872] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10069.883481] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10084.995756] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10093.010425] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10105.826316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10105.835182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10105.861436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10108.907830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10108.912561] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10108.965800] Lustre: Skipped 2 previous similar messages [10111.342921] Lustre: server umount lustre-MDT0000 complete [10115.213988] LustreError: 256230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956717 with bad export cookie 2519386898492928678 [10115.227299] LustreError: 256230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10115.227307] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10115.496306] Lustre: server umount lustre-MDT0001 complete [10128.442616] Lustre: server umount lustre-OST0000 complete [10142.396829] Lustre: server umount lustre-OST0001 complete [10156.626331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10167.714582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10176.902541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10188.366764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10209.308938] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10219.535460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10219.598139] Lustre: lustre-MDT0000: reset Object Index mappings [10220.142625] LustreError: 260471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10220.153822] LustreError: 260471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10220.240248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10225.145022] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10225.643227] LustreError: 260472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10229.738687] LustreError: 260471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.049153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.066656] Lustre: lustre-MDT0001: reset Object Index mappings [10235.410661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10240.295992] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10243.249259] Lustre: 261613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10250.273306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10256.078366] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10261.034806] LustreError: 261966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.045889] LustreError: 261966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10262.464683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10262.686347] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10262.692831] Lustre: Skipped 1 previous similar message [10264.765979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10265.786298] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10267.525925] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10274.067219] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10277.313896] Lustre: 263484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10284.587839] Lustre: server umount lustre-MDT0000 complete [10286.580097] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.595042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10286.596502] LustreError: 260471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10286.596991] Lustre: Skipped 2 previous similar messages [10288.445081] LustreError: 260454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956890 with bad export cookie 2519386898492935468 [10288.456981] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10288.461304] LustreError: 260454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10288.745971] Lustre: server umount lustre-MDT0001 complete [10304.185496] Lustre: server umount lustre-OST0000 complete [10307.808146] Lustre: 250463:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783956893/real 1783956893] req@ffff9eb21276df80 x1870614028744576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783956909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10307.885531] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10307.924854] Lustre: Skipped 1 previous similar message [10309.644748] Lustre: server umount lustre-OST0001 complete [10331.285925] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:35:30 (1783956930) [10351.459474] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [10364.243510] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10402.112245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10402.315153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10402.334342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10402.416888] Lustre: lustre-MDT0000: new disk, initializing [10402.549332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10402.567023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10407.639940] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10419.031040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10419.129424] Lustre: 267741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10419.176783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10419.180711] Lustre: Skipped 1 previous similar message [10419.257924] Lustre: lustre-MDT0001: new disk, initializing [10419.416328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10419.430324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10424.478423] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10434.760746] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10440.311508] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10447.700207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10448.059354] Lustre: lustre-OST0000: new disk, initializing [10448.070831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10448.086149] Lustre: 269711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10449.197311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10449.207194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10449.345552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10454.558992] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10464.394708] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10469.834631] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10470.102746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10474.460577] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10474.642652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10480.098135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10480.104988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10480.110797] LustreError: Skipped 1 previous similar message [10480.111651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10480.111657] Lustre: Skipped 3 previous similar messages [10480.162143] Lustre: Skipped 1 previous similar message [10482.684738] Lustre: server umount lustre-OST0000 complete [10486.240954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10486.254777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10486.263335] Lustre: Skipped 1 previous similar message [10490.337901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10490.339997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.359285] Lustre: Skipped 1 previous similar message [10492.488397] Lustre: server umount lustre-MDT0000 complete [10495.460541] LustreError: 267750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10495.478490] LustreError: 267750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10496.353186] LustreError: 267734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957098 with bad export cookie 2519386898492937561 [10496.363317] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10496.367360] LustreError: 267734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10497.030414] Lustre: server umount lustre-MDT0001 complete [10518.241392] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10529.939930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10530.367824] LustreError: 272643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.423705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10530.433465] Lustre: Skipped 2 previous similar messages [10536.029830] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10545.894133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10546.144986] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10546.154643] Lustre: Skipped 1 previous similar message [10551.619466] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10554.965123] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10561.867273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10568.700842] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10577.992908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10578.263919] Lustre: lustre-OST0001: new disk, initializing [10578.270082] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10578.279087] Lustre: 274951:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10579.907025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10579.921436] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10580.022414] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10585.228775] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10594.410191] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10598.233360] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10603.347485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10603.456664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10610.664391] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.675599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10610.684677] Lustre: Skipped 2 previous similar messages [10610.704323] Lustre: Skipped 3 previous similar messages [10616.606401] Lustre: server umount lustre-MDT0000 complete [10620.833946] LustreError: 272626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957222 with bad export cookie 2519386898492938443 [10620.845335] LustreError: 272626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10620.845809] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10620.897370] LustreError: 274142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.905784] LustreError: 274142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10621.205849] Lustre: server umount lustre-MDT0001 complete [10627.671104] Lustre: server umount lustre-OST0000 complete [10631.950264] Lustre: server umount lustre-OST0001 complete [10649.500587] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10660.390322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10660.840000] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10660.845661] Lustre: Skipped 3 previous similar messages [10665.387661] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10673.994870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.624506] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10682.806815] Lustre: 279147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10689.170600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10695.723915] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10699.709736] LustreError: 279502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10699.737015] LustreError: 279502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10705.934154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10706.101371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10712.071468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10714.970438] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10722.822830] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10726.661549] Lustre: 281029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10733.650501] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10733.871823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10745.621881] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:42:24 (1783957344) [10761.697153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10761.700575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10761.704160] Lustre: Skipped 3 previous similar messages [10761.706894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10761.709209] Lustre: Skipped 6 previous similar messages [10763.860430] Lustre: server umount lustre-MDT0000 complete [10767.186869] LustreError: 279149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957369 with bad export cookie 2519386898492944995 [10767.208826] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10773.554873] Lustre: server umount lustre-MDT0001 complete [10791.394609] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10791.504153] Lustre: server umount lustre-OST0000 complete [10795.109508] Lustre: server umount lustre-OST0001 complete [10801.056443] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [10816.348572] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10867.218144] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10876.909994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10877.170810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10877.216764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10877.306957] Lustre: lustre-MDT0000: new disk, initializing [10877.402647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10881.670401] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10890.824483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10890.958264] Lustre: 285289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10891.172232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10895.100235] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10906.003163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10906.391710] Lustre: 286919:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10908.399672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10913.307762] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10924.335831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10924.522830] Lustre: 287792:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10924.623972] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10924.630178] Lustre: Skipped 6 previous similar messages [10926.487760] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10926.498194] Lustre: Skipped 1 previous similar message [10926.629241] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10932.139828] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10942.129744] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10945.988188] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10945.991621] Lustre: Skipped 1 previous similar message [10962.215864] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:46:01 (1783957561) [10966.916530] Lustre: server umount lustre-MDT0000 complete [10967.530312] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10967.541804] Lustre: Skipped 5 previous similar messages [10967.546369] LustreError: 285294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10967.565905] LustreError: 285294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10970.350117] LustreError: 285280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957572 with bad export cookie 2519386898492952639 [10970.366807] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10970.375170] LustreError: 285280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10970.796847] Lustre: server umount lustre-MDT0001 complete [10984.877786] Lustre: server umount lustre-OST0000 complete [10988.779408] Lustre: 250463:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783957574/real 1783957574] req@ffff9eb20c345880 x1870614029003392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783957590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10989.492267] Lustre: server umount lustre-OST0001 complete [10998.898407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.967868] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11014.395885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11020.504720] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11028.483520] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11034.769534] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11042.356423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11049.572595] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11057.715607] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11077.929428] loop1: detected capacity change from 0 to 204800000 [11078.093917] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11078.112415] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11078.128393] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11083.317121] loop1: detected capacity change from 0 to 204800000 [11088.557576] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11088.665150] LustreError: 290029:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs31367: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11088.678304] LustreError: 290029:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs31367 [11088.683593] LustreError: 290029:0:(mgs_handler.c:517:mgs_target_reg()) fs31367-MDT0001: Error checking target index: rc = -13 [11088.693878] LustreError: fs31367-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11088.701284] LustreError: 293757:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11088.706720] LustreError: 293757:0:(tgt_mount.c:2129:server_put_super()) no obd fs31367-MDT0001 [11088.711977] LustreError: 293757:0:(tgt_mount.c:116:server_deregister_mount()) fs31367-MDT0001 not registered [11088.752714] Lustre: server umount fs31367-MDT0001 complete [11088.757765] LustreError: 293757:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11096.717472] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11096.860462] Lustre: Setting parameter fs31367-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs31367-MDT0001 [11096.929920] Lustre: srv-fs31367-MDT0001: No data found on store. Initialize space. [11096.936942] Lustre: Skipped 4 previous similar messages [11097.029558] Lustre: fs31367-MDT0001: new disk, initializing [11097.035637] Lustre: Skipped 3 previous similar messages [11097.176794] Lustre: srv-fs31367-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11097.186287] Lustre: 294224:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31367-MDT0001-mdtlov: can't create new llog: rc = -115 [11097.197647] LustreError: 294224:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31367-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11101.193484] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11107.158333] Lustre: server umount fs31367-MDT0001 complete [11117.780436] loop1: detected capacity change from 0 to 204800000 [11123.330474] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11123.685193] Lustre: 295791:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31367-MDT0001-mdtlov: can't create new llog: rc = -115 [11123.706354] LustreError: 295791:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31367-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11125.736272] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11128.353391] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11132.424276] Lustre: server umount fs31367-MDT0001 complete [11141.031577] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) [11141.035365] Lustre: Skipped 8 previous similar messages [11141.092198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11141.097089] Lustre: Skipped 5 previous similar messages [11146.684341] Lustre: server umount lustre-MDT0000 complete [11153.141277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11153.296046] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11158.084529] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11159.023954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11165.636288] loop1: detected capacity change from 0 to 204800000 [11171.111719] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11171.541355] Lustre: 298235:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31367-MDT0001-mdtlov: can't create new llog: rc = -115 [11171.554792] LustreError: 298235:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31367-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11175.710267] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11179.609671] Lustre: server umount fs31367-MDT0001 complete [11199.977517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11199.986086] LustreError: Skipped 1 previous similar message [11203.697988] Lustre: server umount lustre-OST0000 complete [11213.200307] Lustre: server umount lustre-MDT0000 complete [11216.403857] LustreError: 296283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957818 with bad export cookie 2519386898492955320 [11216.407541] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11216.419263] LustreError: 296283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11216.689129] Lustre: server umount lustre-MDT0001 complete [11226.681737] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11228.896680] Key type lgssc unregistered [11229.185903] LNet: 300578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11229.191555] LNetError: 300578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11230.248335] LNet: Removed LNI 192.168.203.109@tcp [11231.019168] Key type .llcrypt unregistered [11231.020369] Key type ._llcrypt unregistered [11248.806378] Key type ._llcrypt registered [11248.809132] Key type .llcrypt registered [11248.913603] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [11249.836536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11249.916231] alg: No test for adler32 (adler32-zlib) [11251.024253] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [11251.232349] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [11252.888194] Key type lgssc registered [11254.172520] Lustre: Echo OBD driver; http://www.lustre.org/ [11261.531374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11261.557864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11262.961031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11267.000716] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11272.169965] LustreError: 301851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11272.204394] LustreError: 301851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11274.709919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.389616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11280.372169] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11288.694841] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11295.369024] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11302.402276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11302.889991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11308.008735] LustreError: 303680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.986040] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11313.151626] LustreError: 303681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11315.608490] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11318.193620] LustreError: 303679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11323.277800] LustreError: 303986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11323.316833] LustreError: 303986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11333.527231] LustreError: 303681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11333.542739] LustreError: 303681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11336.070771] loop1: detected capacity change from 0 to 204800000 [11336.306218] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11336.325238] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11336.335771] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11341.536180] loop1: detected capacity change from 0 to 204800000 [11346.896526] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11347.039891] Lustre: fs5371-OST0000: new disk, initializing [11347.044788] Lustre: srv-fs5371-OST0000: No data found on store. Initialize space. [11347.052037] Lustre: 305587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11347.124100] Lustre: fs5371-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11351.012936] LustreError: 303679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11351.029762] LustreError: 303679:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11352.513597] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11360.355752] Lustre: server umount fs5371-OST0000 complete [11371.052736] loop1: detected capacity change from 0 to 204800000 [11376.067267] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11376.280169] Lustre: fs5371-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11381.740291] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11381.841792] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11384.729138] LustreError: 305226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11384.748903] LustreError: 305226:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11390.146155] loop2: detected capacity change from 0 to 204800000 [11390.321833] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11390.329427] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11390.341094] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11395.084679] loop2: detected capacity change from 0 to 204800000 [11400.705987] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11400.801990] LustreError: fs5371-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11400.814661] LustreError: fs5371-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11400.821613] LustreError: 308625:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11400.827898] LustreError: 308625:0:(tgt_mount.c:2129:server_put_super()) no obd fs5371-OST0001 [11400.832550] LustreError: 308625:0:(tgt_mount.c:116:server_deregister_mount()) fs5371-OST0001 not registered [11400.868669] Lustre: server umount fs5371-OST0001 complete [11400.873076] LustreError: 308625:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11402.218099] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11411.573276] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11411.779624] Lustre: fs5371-OST0001: new disk, initializing [11411.783492] Lustre: srv-fs5371-OST0001: No data found on store. Initialize space. [11411.790724] Lustre: 309176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11411.892445] Lustre: fs5371-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11417.302884] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11425.222509] Lustre: server umount fs5371-OST0000 complete [11434.283341] Lustre: server umount fs5371-OST0001 complete [11446.551369] loop1: detected capacity change from 0 to 204800000 [11451.277889] LustreError: 303986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11451.308068] LustreError: 303986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11451.846977] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11452.042298] Lustre: fs5371-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11458.075807] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11464.509760] loop2: detected capacity change from 0 to 204800000 [11469.842366] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11476.118848] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11485.406919] Lustre: server umount fs5371-OST0000 complete [11494.772245] Lustre: server umount fs5371-OST0001 complete [11512.802048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11512.811501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.837726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11513.825378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.856350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11520.000563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11520.004451] Lustre: Skipped 3 previous similar messages [11524.068262] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11524.309037] Lustre: server umount lustre-OST0000 complete [11530.213751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.214993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11530.224452] Lustre: Skipped 1 previous similar message [11533.850491] Lustre: server umount lustre-MDT0000 complete [11537.209596] LustreError: 313593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958139 with bad export cookie 3736531618986944263 [11537.221060] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11537.226582] LustreError: 313593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11537.660165] Lustre: server umount lustre-MDT0001 complete [11548.572235] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11551.464831] Key type lgssc unregistered [11551.786179] LNet: 315100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11551.794936] LNetError: 315100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11551.836218] LNet: Removed LNI 192.168.203.109@tcp [11552.766665] Key type .llcrypt unregistered [11552.771661] Key type ._llcrypt unregistered [11583.795613] Key type ._llcrypt registered [11583.798490] Key type .llcrypt registered [11583.912258] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [11600.433547] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [11601.428284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11601.495520] alg: No test for adler32 (adler32-zlib) [11602.569397] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [11602.755830] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [11604.411497] Key type lgssc registered [11605.398826] Lustre: Echo OBD driver; http://www.lustre.org/ [11640.085680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11640.118801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11641.314319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11641.338361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11641.452408] Lustre: lustre-MDT0000: new disk, initializing [11641.529688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11641.548660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11645.731962] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11654.636044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11654.739373] Lustre: 318960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11654.789620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11654.793101] Lustre: Skipped 1 previous similar message [11654.856661] Lustre: lustre-MDT0001: new disk, initializing [11654.965989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11655.017785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11655.026197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11658.381932] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11666.131503] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11670.784596] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11676.257407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11676.484333] Lustre: lustre-OST0000: new disk, initializing [11676.489076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11676.493464] Lustre: 320929:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11676.601466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11683.134391] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11685.517262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11685.531407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11685.607820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11691.670840] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11693.925600] LustreError: 317133:0:(lmv_obd.c:208:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [11693.984603] Lustre: Mounted lustre-client [11695.127936] Lustre: Unmounted lustre-client [11706.360580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11706.361571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11706.383594] LustreError: Skipped 1 previous similar message [11706.387761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11706.422249] Lustre: Skipped 1 previous similar message [11706.446098] Lustre: Skipped 1 previous similar message [11709.416756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11709.424454] Lustre: Skipped 1 previous similar message [11710.804691] Lustre: server umount lustre-OST0000 complete [11714.536600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11714.540323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11714.554172] Lustre: Skipped 1 previous similar message [11714.580038] Lustre: Skipped 1 previous similar message [11719.649507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11719.670109] Lustre: Skipped 1 previous similar message [11720.694997] Lustre: server umount lustre-MDT0000 complete [11723.957577] LustreError: 318953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958325 with bad export cookie 12453445183496565291 [11723.967161] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11723.969821] LustreError: 318953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11724.234993] Lustre: server umount lustre-MDT0001 complete [11733.566464] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11735.723205] Key type lgssc unregistered [11745.120506] LNet: 1 local NIs in recovery (showing 1): 192.168.203.109@tcp [11745.190377] LNet: 323086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11745.195871] LNetError: 323086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11827.116292] LNet: Removed LNI 192.168.203.109@tcp [11827.956173] Key type .llcrypt unregistered [11827.958774] Key type ._llcrypt unregistered [11845.101489] Key type ._llcrypt registered [11845.105494] Key type .llcrypt registered [11845.235962] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [11862.779551] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [11863.973671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11863.993735] alg: No test for adler32 (adler32-zlib) [11865.294802] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [11865.532531] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [11867.209848] Key type lgssc registered [11868.816142] Lustre: Echo OBD driver; http://www.lustre.org/ [11903.247486] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:01:42 (1783958502) [11909.983531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11910.006144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11911.233489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11911.276386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11911.367141] Lustre: lustre-MDT0000: new disk, initializing [11911.460765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11911.494957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11916.347168] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11926.289833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11926.401267] Lustre: 326320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11926.492952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11926.500867] Lustre: Skipped 1 previous similar message [11926.609177] Lustre: lustre-MDT0001: new disk, initializing [11926.707750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11926.755058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11926.774534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11931.776279] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11940.036829] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11945.583878] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11953.557973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11953.807593] Lustre: lustre-OST0000: new disk, initializing [11953.821399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11953.830718] Lustre: 328290:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11953.918696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11960.446077] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11964.005651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11964.025306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11964.208866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11969.992553] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11984.865881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11984.871630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11984.884017] Lustre: Skipped 1 previous similar message [11984.887453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11984.890599] LustreError: Skipped 1 previous similar message [11984.914828] Lustre: Skipped 1 previous similar message [11989.990758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11989.998626] Lustre: Skipped 1 previous similar message [11990.189542] Lustre: server umount lustre-OST0000 complete [11993.578230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11993.595546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11993.611592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11995.109736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11999.795351] Lustre: server umount lustre-MDT0000 complete [12003.145070] LustreError: 329158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958604 with bad export cookie 12347291227847102156 [12003.152244] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12003.154424] LustreError: 329158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12003.424602] Lustre: server umount lustre-MDT0001 complete [12013.988699] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12016.915381] Key type lgssc unregistered [12017.254971] LNet: 330198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12017.269487] LNetError: 330198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12018.341769] LNet: Removed LNI 192.168.203.109@tcp [12019.449657] Key type .llcrypt unregistered [12019.453618] Key type ._llcrypt unregistered [12039.382947] Key type ._llcrypt registered [12039.385620] Key type .llcrypt registered [12039.488958] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12040.376259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12040.402997] alg: No test for adler32 (adler32-zlib) [12041.572348] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [12041.906554] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [12043.568473] Key type lgssc registered [12044.532414] Lustre: Echo OBD driver; http://www.lustre.org/ [12052.316807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12052.336131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12053.754660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12058.775947] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12063.206497] LustreError: 331472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12063.219574] LustreError: 331472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12068.047442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.323479] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12068.450420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12073.043956] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12080.858916] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12087.068559] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12094.234883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.695470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12100.507836] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12108.710343] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12123.967685] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12125.847344] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12129.763085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12129.764244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12129.766086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12129.803617] Lustre: Skipped 2 previous similar messages [12132.257907] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) [12132.262791] Lustre: Skipped 2 previous similar messages [12134.882954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12134.898776] Lustre: Skipped 2 previous similar messages [12140.014465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12140.019352] Lustre: Skipped 3 previous similar messages [12144.096127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12144.262602] Lustre: server umount lustre-MDT0000 complete [12146.578497] LustreError: 331453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958748 with bad export cookie 17357500228789593087 [12146.583608] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12147.588093] LustreError: 331467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12148.196381] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12148.206931] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12148.210913] Lustre: Skipped 6 previous similar messages [12151.200812] LustreError: 331468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12151.208105] LustreError: 331468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12152.805404] Lustre: server umount lustre-MDT0001 complete [12154.683404] Lustre: server umount lustre-OST0000 complete [12161.908290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12167.254353] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12170.354459] Lustre: 335651:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.109@tcp [12172.968586] Lustre: server umount MGS complete [12182.939661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12183.290598] LustreError: 336222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12183.301038] LustreError: 336222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12183.415385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12188.397315] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12188.643069] LustreError: 336223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12198.618757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12198.883249] LustreError: 336223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12198.907096] LustreError: 336223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12199.373810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12204.082238] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12213.395773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12213.701993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12220.495963] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12229.505587] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12231.393361] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12237.757302] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12239.580684] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12245.474716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12245.486059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.511067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12245.519234] Lustre: Skipped 2 previous similar messages [12246.913562] Lustre: server umount lustre-OST0000 complete [12254.693182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12254.706269] Lustre: Skipped 2 previous similar messages [12257.257725] Lustre: server umount lustre-MDT0000 complete [12259.815352] LustreError: 336219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12259.845484] LustreError: 336219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12261.176122] LustreError: 336202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958863 with bad export cookie 17357500228789593934 [12261.182861] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12261.191489] LustreError: 336202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12261.515015] Lustre: server umount lustre-MDT0001 complete [12272.003182] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12274.678667] Key type lgssc unregistered [12274.956894] LNet: 339659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12274.966224] LNetError: 339659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12274.995311] LNet: Removed LNI 192.168.203.109@tcp [12275.941276] Key type .llcrypt unregistered [12275.947460] Key type ._llcrypt unregistered [12294.822107] Key type ._llcrypt registered [12294.828095] Key type .llcrypt registered [12294.932398] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [12312.732584] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12313.935207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12313.972848] alg: No test for adler32 (adler32-zlib) [12315.022507] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [12315.357475] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [12317.113613] Key type lgssc registered [12318.346669] Lustre: Echo OBD driver; http://www.lustre.org/ [12353.758906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12353.775098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12355.056202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12355.086813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12355.189468] Lustre: lustre-MDT0000: new disk, initializing [12355.302394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12355.330243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12359.601268] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12370.639816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12370.808408] Lustre: 342823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12370.882853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12370.886741] Lustre: Skipped 1 previous similar message [12371.008629] Lustre: lustre-MDT0001: new disk, initializing [12371.163090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12371.219267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12371.243532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12375.913903] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12385.154971] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12390.948945] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12397.389874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12397.734219] Lustre: lustre-OST0000: new disk, initializing [12397.738051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12397.744779] Lustre: 344791:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12397.869532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12403.781039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12403.789591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12403.910854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12405.330194] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12415.152246] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12419.577921] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12419.761549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12424.024857] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12424.273675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12429.793755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12429.806361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12429.847033] Lustre: Skipped 1 previous similar message [12429.855343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12432.542813] Lustre: server umount lustre-OST0000 complete [12438.011187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12438.018129] LustreError: Skipped 1 previous similar message [12438.023709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12438.042439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12438.057454] Lustre: Skipped 1 previous similar message [12440.035229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12440.035902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12442.249177] Lustre: server umount lustre-MDT0000 complete [12445.758485] LustreError: 342816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783959047 with bad export cookie 14022523762503805160 [12445.768484] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12445.772698] LustreError: 342816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12446.058579] Lustre: server umount lustre-MDT0001 complete [12455.119001] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:10:54 (1783959054) [12470.790728] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [12480.357560] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12518.070154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12518.377597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12518.410845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12518.529277] Lustre: lustre-MDT0000: new disk, initializing [12518.660749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12518.688683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12522.549577] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12532.178691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12532.296481] Lustre: 349551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12532.340502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12532.344387] Lustre: Skipped 1 previous similar message [12532.422097] Lustre: lustre-MDT0001: new disk, initializing [12532.506098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12532.537280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12532.550077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12536.330632] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12545.536311] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12551.010834] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12557.508537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12557.838707] Lustre: lustre-OST0000: new disk, initializing [12557.845103] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12557.862777] Lustre: 351518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12557.986299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12564.627431] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12567.658164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12567.671626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12567.756185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12574.209405] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12614.117435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12614.118227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12614.118565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12614.118570] Lustre: Skipped 1 previous similar message [12614.124402] Lustre: Skipped 1 previous similar message [12616.745429] Lustre: server umount lustre-OST0000 complete [12624.355118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12624.360603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12624.368892] Lustre: Skipped 1 previous similar message [12624.384752] Lustre: Skipped 2 previous similar messages [12626.868615] Lustre: server umount lustre-MDT0000 complete [12629.475138] LustreError: 349562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12629.502885] LustreError: 349562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12631.384867] LustreError: 350354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783959233 with bad export cookie 14022523762503806042 [12631.387394] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12631.411049] LustreError: 350354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12631.816450] Lustre: server umount lustre-MDT0001 complete [12644.840596] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12647.277495] Key type lgssc unregistered [12647.571941] LNet: 353692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12647.584578] LNetError: 353692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12647.595420] LNet: Removed LNI 192.168.203.109@tcp [12648.420049] Key type .llcrypt unregistered [12648.422805] Key type ._llcrypt unregistered [12742.091135] Key type ._llcrypt registered [12742.094887] Key type .llcrypt registered [12742.214547] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12742.529540] Key type .llcrypt unregistered [12742.531213] Key type ._llcrypt unregistered [12821.876221] Key type ._llcrypt registered [12821.881714] Key type .llcrypt registered [12822.003530] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12822.286574] Key type .llcrypt unregistered [12822.292104] Key type ._llcrypt unregistered [12854.240783] Key type ._llcrypt registered [12854.247775] Key type .llcrypt registered [12854.356198] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12854.603532] Key type .llcrypt unregistered [12854.607200] Key type ._llcrypt unregistered [12886.452633] Key type ._llcrypt registered [12886.454390] Key type .llcrypt registered [12886.629889] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12887.079372] Key type .llcrypt unregistered [12887.084667] Key type ._llcrypt unregistered [12965.683263] Key type ._llcrypt registered [12965.686117] Key type .llcrypt registered [12965.823776] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12966.121607] Key type .llcrypt unregistered [12966.123763] Key type ._llcrypt unregistered [12999.053321] Key type ._llcrypt registered [12999.054976] Key type .llcrypt registered [12999.196737] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12999.515348] Key type .llcrypt unregistered [12999.525412] Key type ._llcrypt unregistered [13036.123147] Key type ._llcrypt registered [13036.124814] Key type .llcrypt registered [13036.238262] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [13036.386300] Key type .llcrypt unregistered [13036.388154] Key type ._llcrypt unregistered [13066.369738] Key type ._llcrypt registered [13066.371836] Key type .llcrypt registered [13066.459912] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [13066.713335] Key type .llcrypt unregistered [13066.717205] Key type ._llcrypt unregistered [13101.460494] Key type ._llcrypt registered [13101.462811] Key type .llcrypt registered [13101.597539] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [13101.803486] Key type .llcrypt unregistered [13101.805307] Key type ._llcrypt unregistered [13132.365933] Key type ._llcrypt registered [13132.369607] Key type .llcrypt registered [13132.496426] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [13132.752683] Key type .llcrypt unregistered [13132.755674] Key type ._llcrypt unregistered [13200.791071] Key type ._llcrypt registered [13200.794108] Key type .llcrypt registered [13200.880701] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [13201.079673] Key type .llcrypt unregistered [13201.083918] Key type ._llcrypt unregistered [13232.913470] Key type ._llcrypt registered [13232.920092] Key type .llcrypt registered [13233.056908] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [13233.438322] Key type .llcrypt unregistered [13233.443384] Key type ._llcrypt unregistered [13267.957342] Key type ._llcrypt registered [13267.962436] Key type .llcrypt registered [13268.089172] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [13268.456357] Key type .llcrypt unregistered [13268.461383] Key type ._llcrypt unregistered [13325.423141] Key type ._llcrypt registered [13325.425984] Key type .llcrypt registered [13325.556085] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [13341.857301] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [13342.535710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13342.569512] alg: No test for adler32 (adler32-zlib) [13343.698956] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [13343.960968] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [13345.632161] Key type lgssc registered [13346.498842] Lustre: Echo OBD driver; http://www.lustre.org/ [13389.967680] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13390.012738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13391.213678] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13391.234465] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13391.297837] Lustre: tst250fs-MDT0000: new disk, initializing [13391.386894] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13391.405486] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13395.770088] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [13410.192257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13410.340409] Lustre: 374037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13410.422444] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13410.427359] Lustre: Skipped 1 previous similar message [13410.578348] Lustre: tst250fs-MDT0001: new disk, initializing [13410.687435] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13410.726393] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13410.745434] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13416.231794] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [13421.729061] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13428.935265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13429.286726] Lustre: tst250fs-OST0000: new disk, initializing [13429.290044] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13429.295831] Lustre: 375670:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13429.401313] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13436.667772] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [13438.014920] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13438.037331] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13438.193538] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13449.325325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13449.611129] Lustre: tst250fs-OST0001: new disk, initializing [13449.625628] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13449.640210] Lustre: 376543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13449.810867] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13454.957550] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13454.971412] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13455.117447] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13456.773658] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [13462.792079] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13465.601143] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13465.603850] Lustre: Skipped 2 previous similar messages [13486.052514] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13486.055702] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13486.069891] Lustre: Skipped 2 previous similar messages [13486.083027] Lustre: Skipped 3 previous similar messages [13491.170759] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13491.187200] Lustre: Skipped 3 previous similar messages [13491.749846] Lustre: server umount tst250fs-MDT0000 complete [13495.799746] LustreError: 374833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783960097 with bad export cookie 9640334297652871117 [13495.812184] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13495.818995] LustreError: 374833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13496.316964] Lustre: server umount tst250fs-MDT0001 complete [13511.051758] Lustre: server umount tst250fs-OST0000 complete [13512.481100] Lustre: 372258:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783960098/real 1783960098] req@ffff9eb30595ad80 x1870617583362176/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783960114 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13512.510863] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13512.532957] Lustre: Skipped 1 previous similar message [13515.115295] Lustre: server umount tst250fs-OST0001 complete [13534.437400] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [13544.297799] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [13580.505823] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:29:39 (1783960179) [13582.372695] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13602.494882] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [13613.281814] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [13647.938171] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13396 sec ========= 12:30:47 (1783960247) [13650.076111] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:30:49 (1783960249) === [13653.598404] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:30:52 (1783960252) === [13692.117526] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [13695.160672] Key type lgssc unregistered [13695.523522] LNet: 385687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13695.534485] LNetError: 385687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13695.554116] LNet: Removed LNI 192.168.203.109@tcp [13696.400163] Key type .llcrypt unregistered [13696.401816] Key type ._llcrypt unregistered