[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 581812709 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003279] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.011074] LSM: Security Framework initializing [ 0.012039] Yama: becoming mindful. [ 0.013025] SELinux: Initializing. [ 0.014049] *** VALIDATE selinux *** [ 0.021712] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026650] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029096] *** VALIDATE tmpfs *** [ 0.031251] *** VALIDATE proc *** [ 0.032206] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034234] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037023] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041067] debug: unmapping init [mem 0xffffffff8d459000-0xffffffff8d460fff] [ 0.043264] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046019] ... bit width: 48 [ 0.047009] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049078] ... max period: 00007fffffffffff [ 0.050011] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052354] rcu: Hierarchical SRCU implementation. [ 0.054614] smp: Bringing up secondary CPUs ... [ 0.055572] x86: Booting SMP configuration: [ 0.056021] .... node #0, CPUs: #1 #2 #3 [ 0.064029] smp: Brought up 1 node, 4 CPUs [ 0.065929] smpboot: Max logical packages: 1 [ 0.066010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125341] node 0 deferred pages initialised in 56ms [ 0.131103] devtmpfs: initialized [ 0.132179] x86/mm: Memory block size: 128MB [ 0.135619] gcov: version magic: 0x41383552 [ 0.137249] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.138058] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139213] pinctrl core: initialized pinctrl subsystem [ 0.140183] [ 0.140939] ************************************************************* [ 0.141015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142011] ** ** [ 0.143007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.144009] ** ** [ 0.145009] ** This means that this kernel is built to expose internal ** [ 0.146012] ** IOMMU data structures, which may compromise security on ** [ 0.147009] ** your system. ** [ 0.148012] ** ** [ 0.149009] ** If you see this message and you are not debugging the ** [ 0.150009] ** kernel, report this immediately to your vendor! ** [ 0.151009] ** ** [ 0.152007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153005] ************************************************************* [ 0.154600] NET: Registered protocol family 16 [ 0.155459] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.156046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.157049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.158818] cpuidle: using governor menu [ 0.159893] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.160467] PCI: Using configuration type 1 for base access [ 0.161145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.170778] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.171000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.173072] cryptd: max_cpu_qlen set to 1000 [ 0.175031] ACPI: Added _OSI(Module Device) [ 0.176021] ACPI: Added _OSI(Processor Device) [ 0.177008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.178006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.183591] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.187277] ACPI: Interpreter enabled [ 0.188077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.189010] ACPI: Using IOAPIC for interrupt routing [ 0.190094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.191426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.205027] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.206034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.207018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.208073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.210523] acpiphp: Slot [2] registered [ 0.211085] acpiphp: Slot [5] registered [ 0.212086] acpiphp: Slot [6] registered [ 0.213208] acpiphp: Slot [7] registered [ 0.214085] acpiphp: Slot [8] registered [ 0.215085] acpiphp: Slot [9] registered [ 0.216079] acpiphp: Slot [10] registered [ 0.217087] acpiphp: Slot [3] registered [ 0.218105] acpiphp: Slot [4] registered [ 0.219102] acpiphp: Slot [11] registered [ 0.220073] acpiphp: Slot [12] registered [ 0.221079] acpiphp: Slot [13] registered [ 0.222217] acpiphp: Slot [14] registered [ 0.223089] acpiphp: Slot [15] registered [ 0.224073] acpiphp: Slot [16] registered [ 0.225071] acpiphp: Slot [17] registered [ 0.226090] acpiphp: Slot [18] registered [ 0.227077] acpiphp: Slot [19] registered [ 0.228082] acpiphp: Slot [20] registered [ 0.229083] acpiphp: Slot [21] registered [ 0.230088] acpiphp: Slot [22] registered [ 0.231243] acpiphp: Slot [23] registered [ 0.232107] acpiphp: Slot [24] registered [ 0.233082] acpiphp: Slot [25] registered [ 0.234102] acpiphp: Slot [26] registered [ 0.235096] acpiphp: Slot [27] registered [ 0.236091] acpiphp: Slot [28] registered [ 0.237075] acpiphp: Slot [29] registered [ 0.238091] acpiphp: Slot [30] registered [ 0.239114] acpiphp: Slot [31] registered [ 0.240058] PCI host bridge to bus 0000:00 [ 0.241016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.242019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.243015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.244017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.245017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.246017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.247165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.249045] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.251232] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.257012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.260334] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.261015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.262013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.263023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.264662] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.265723] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.266041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.267608] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.273013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.286965] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.292012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.304246] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.313013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.322017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.344014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.354000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.365015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.375014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.395012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.406000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.412016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.423013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.457014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.476842] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.489117] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.500011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.529013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.545561] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.552012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.558013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.600484] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.612017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.622013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.636015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.646773] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.650396] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.653449] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.656427] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.658347] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.663023] iommu: Default domain type: Passthrough [ 0.665000] SCSI subsystem initialized [ 0.667146] ACPI: bus type USB registered [ 0.669121] usbcore: registered new interface driver usbfs [ 0.671067] usbcore: registered new interface driver hub [ 0.673067] usbcore: registered new device driver usb [ 0.675209] pps_core: LinuxPPS API ver. 1 registered [ 0.677009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.681066] PTP clock support registered [ 0.683250] EDAC MC: Ver: 3.0.0 [ 0.685924] PCI: Using ACPI for IRQ routing [ 0.686982] NetLabel: Initializing [ 0.688010] NetLabel: domain hash size = 128 [ 0.690011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.693135] NetLabel: unlabeled traffic allowed by default [ 0.695354] vgaarb: loaded [ 0.697845] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.700015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.707464] clocksource: Switched to clocksource kvm-clock [ 0.822243] VFS: Disk quotas dquot_6.6.0 [ 0.823382] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.825995] *** VALIDATE ramfs *** [ 0.827595] *** VALIDATE hugetlbfs *** [ 0.829563] pnp: PnP ACPI init [ 0.833153] pnp: PnP ACPI: found 6 devices [ 0.851794] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.856525] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.859021] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.861980] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.864579] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.866876] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.870388] NET: Registered protocol family 2 [ 0.875066] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.896112] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.902093] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.910752] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.923516] TCP: Hash tables configured (established 65536 bind 65536) [ 0.928121] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.932807] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.938115] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953050] NET: Registered protocol family 1 [ 1.003522] RPC: Registered named UNIX socket transport module. [ 1.005547] RPC: Registered udp transport module. [ 1.006814] RPC: Registered tcp transport module. [ 1.008256] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.010192] NET: Registered protocol family 44 [ 1.011353] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.013173] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.014759] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.017211] PCI: CLS 0 bytes, default 64 [ 1.018967] Unpacking initramfs... [ 3.761199] debug: unmapping init [mem 0xffff94347cc54000-0xffff94347ffbffff] [ 3.782160] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.799563] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.805274] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.983305] Initialise system trusted keyrings [ 4.984971] Key type blacklist registered [ 4.986932] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.996357] zbud: loaded [ 4.999215] *** VALIDATE nfs *** [ 5.000520] *** VALIDATE nfs4 *** [ 5.001810] pstore: using deflate compression [ 5.006150] Platform Keyring initialized [ 5.310508] NET: Registered protocol family 38 [ 5.317398] Key type asymmetric registered [ 5.321435] Asymmetric key parser 'x509' registered [ 5.326758] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.338575] io scheduler mq-deadline registered [ 5.341266] io scheduler kyber registered [ 5.349848] io scheduler bfq registered [ 5.357486] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.372097] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.386628] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.396967] ACPI: Power Button [PWRF] [ 5.432960] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.462076] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.487330] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.583302] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.610043] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.653980] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.683913] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.688405] Non-volatile memory driver v1.3 [ 5.689702] Linux agpgart interface v0.103 [ 5.804347] virtio_blk virtio1: [vda] 146752 512-byte logical blocks (75.1 MB/71.7 MiB) [ 5.817083] vda: detected capacity change from 0 to 75137024 [ 5.868781] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.871839] vdb: detected capacity change from 0 to 1073741824 [ 5.892209] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.895682] vdc: detected capacity change from 0 to 2621440000 [ 5.911488] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.915015] vdd: detected capacity change from 0 to 2621440000 [ 5.942476] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.952207] vde: detected capacity change from 0 to 4294967296 [ 5.995837] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.000426] vdf: detected capacity change from 0 to 4294967296 [ 6.011506] libphy: Fixed MDIO Bus: probed [ 6.019945] usbcore: registered new interface driver usbserial_generic [ 6.023625] usbserial: USB Serial support registered for generic [ 6.028377] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.043384] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.051639] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.062509] mousedev: PS/2 mouse device common for all mice [ 6.081086] rtc_cmos 00:05: RTC can wake from S4 [ 6.092808] rtc_cmos 00:05: registered as rtc0 [ 6.097571] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.109796] intel_pstate: CPU model not supported [ 6.117522] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.133185] hid: raw HID events driver (C) Jiri Kosina [ 6.138937] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.142629] usbcore: registered new interface driver usbhid [ 6.142637] usbhid: USB HID core driver [ 6.142758] drop_monitor: Initializing network drop monitor service [ 6.142907] Initializing XFRM netlink socket [ 6.143310] NET: Registered protocol family 10 [ 6.157567] Segment Routing with IPv6 [ 6.162397] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.164822] NET: Registered protocol family 17 [ 6.165949] mpls_gso: MPLS GSO support [ 6.226149] RAS: Correctable Errors collector initialized. [ 6.236812] AVX version of gcm_enc/dec engaged. [ 6.244909] AES CTR mode by8 optimization enabled [ 6.513174] sched_clock: Marking stable (6513150815, 0)->(7890502829, -1377352014) [ 6.517136] registered taskstats version 1 [ 6.518726] Loading compiled-in X.509 certificates [ 6.522402] zswap: loaded using pool lzo/zbud [ 6.562411] Key type big_key registered [ 6.582317] Key type encrypted registered [ 6.583498] ima: No TPM chip found, activating TPM-bypass! [ 6.585468] ima: Allocated hash algorithm: sha1 [ 6.586983] ima: No architecture policies found [ 6.588332] evm: Initialising EVM extended attributes: [ 6.589609] evm: security.selinux [ 6.590583] evm: security.ima [ 6.591420] evm: security.capability [ 6.592455] evm: HMAC attrs: 0x1 [ 6.594450] rtc_cmos 00:05: setting system clock to 2026-09-08 12:03:47 UTC (1788869027) [ 6.604623] debug: unmapping init [mem 0xffffffff8e403000-0xffffffff8e5fffff] [ 6.607679] debug: unmapping init [mem 0xffffffff8d182000-0xffffffff8d458fff] [ 6.618966] Write protecting the kernel read-only data: 28672k [ 6.626938] debug: unmapping init [mem 0xffffffff8b803000-0xffffffff8b9fffff] [ 6.633228] debug: unmapping init [mem 0xffffffff8c114000-0xffffffff8c1fffff] [ 6.690702] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.697809] systemd[1]: Detected virtualization kvm. [ 6.699478] systemd[1]: Detected architecture x86-64. [ 6.701198] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.763474] systemd[1]: No hostname configured. [ 6.767355] systemd[1]: Set hostname to . [ 6.771752] random: systemd: uninitialized urandom read (16 bytes read) [ 6.775342] systemd[1]: Initializing machine ID from random generator. [ 7.093121] random: ln: uninitialized urandom read (6 bytes read) [ 7.427721] random: systemd: uninitialized urandom read (16 bytes read) [ 7.434191] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 7.443143] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.448787] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.958631] device-mapper: uevent: version 1.0.3 [ 9.981413] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.788078] virtio_net virtio0 ens2: renamed from eth0 [ 12.962888] random: fast init done [ 13.186846] scsi host0: ata_piix [ 13.317329] scsi host1: ata_piix [ 13.320103] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.327110] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.512877] random: crng init done [ 18.532528] random: 7 urandom warning(s) missed due to ratelimiting [ 21.835292] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 26.630337] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 35.491414] printk: systemd: 26 output lines suppressed due to ratelimiting [ 37.013543] SELinux: Disabled at runtime. [ 37.198295] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 37.217166] systemd[1]: Detected virtualization kvm. [ 37.219735] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 43.254466] systemd[1]: initrd-switch-root.service: Succeeded. [ 43.357044] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 43.388532] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 43.411659] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 43.422736] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 43.475495] systemd[1]: Starting Journal Service... Starting Journal Service... [ 43.516891] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 43.545921] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ 43.588962] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 43.660402] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ 43.983908] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). [ 46.197235] hrtimer: interrupt took 5226355 ns Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 47.623253] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 50.465337] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 50.681777] input: PC Speaker as /devices/platform/pcspkr/input/input5 [[ 54.918864] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 55.697226] EDAC sbridge: Ver: 1.1.2 * ] A start job is running for Configur…only root support (11s / no limit) [** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [ ***] A start job is running for Configur…only root support (16s / no limit) [ **] A start job is running for Configur…only root support (17s / no limit) [ *] A start job is running for Configur…only root support (17s / no limit) [ **] A start job is running for Configur…only root support (18s / no limit) [ ***] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit) [ *** ] A start job is running for Configur…only root support (20s / no limit) [*** ] A start job is running for Configur…only root support (20s / no limit) [** ] A start job is running for Configur…only root support (21s / no limit) [* ] A start job is running for Configur…only root support (21s / no limit) [** ] A start job is running for Configur…only root support (22s / no limit) [*** ] A start job is running for Configur…only root support (22s / no limit) [ *** ] A start job is running for Configur…only root support (23s / no limit) [ *** ] A start job is running for Configur…only root support (24s / no limit) [ ***] A start job is running for Configur…only root support (24s / no limit) [ **] A start job is running for Configur…only root support (25s / no limit)[ 68.572814] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (25s / no limit) [ **] A start job is running for Configur…only root support (26s / no limit) [ ***] A start job is running for Configur…only root support (26s / no limit) [ *** ] A start job is running for Configur…only root support (27s / no limit)[ 70.365949] NFS: Registering the id_resolver key type [ 70.379852] Key type id_resolver registered [ 70.381461] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (27s / no limit) [*** ] A start job is running for Configur…only root support (28s / no limit) [** ] A start job is running for Configur…only root support (28s / no limit) [* ] A start job is running for Configur…only root support (29s / no limit) [** ] A start job is running for Configur…only root support (29s / no limit) [*** ] A start job is running for Configur…only root support (30s / no limit) [ *** ] A start job is running for Configur…only root support (30s / no limit) [ *** ] A start job is running for Configur…only root support (31s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg347-server login: [ 133.603279] libcfs: loading out-of-tree module taints kernel. [ 133.676945] Key type ._llcrypt registered [ 133.679984] Key type .llcrypt registered [ 133.785958] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 151.865951] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 153.712692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 153.724695] alg: No test for adler32 (adler32-zlib) [ 155.067759] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 155.940358] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 157.703165] Key type lgssc registered [ 159.495285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 178.388265] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 223.307655] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 242.703282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 242.758558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 243.968130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 244.022699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 244.096879] Lustre: lustre-MDT0000: new disk, initializing [ 244.164229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.180786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 248.674273] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.512510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 262.593833] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 262.630113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 262.634195] Lustre: Skipped 1 previous similar message [ 262.715803] Lustre: lustre-MDT0001: new disk, initializing [ 262.817116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 262.863294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 262.870761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 267.403325] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 272.388055] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 282.110121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 282.420604] Lustre: lustre-OST0000: new disk, initializing [ 282.425205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 282.432666] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 282.503087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 287.245974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 287.271089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 287.356961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 289.200643] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 305.198669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 305.322426] Lustre: lustre-OST0001: new disk, initializing [ 305.326719] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 305.333134] Lustre: 9527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 305.399872] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 311.739639] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 313.390793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 313.405090] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 313.467833] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 324.004986] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 334.926534] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 343.009577] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing check_logdir /tmp/testlogs/ [ 348.546119] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing yml_node [ 353.293644] Lustre: DEBUG MARKER: Client: 2.17.57.117 [ 356.339760] Lustre: DEBUG MARKER: MDS: 2.17.57.117 [ 359.313559] Lustre: DEBUG MARKER: OSS: 2.17.57.117 [ 361.219625] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 08:09:40 EDT 2026 [ 381.443567] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 383.190497] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 400.353083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.357125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 400.373516] Lustre: Skipped 3 previous similar messages [ 400.383470] Lustre: Skipped 3 previous similar messages [ 405.472338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 405.481569] Lustre: Skipped 3 previous similar messages [ 406.160544] Lustre: server umount lustre-MDT0000 complete [ 410.015207] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869430 with bad export cookie 13299436217861615524 [ 410.016077] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 410.041214] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 410.597074] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.610194] Lustre: Skipped 1 previous similar message [ 410.619754] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.628282] Lustre: Skipped 1 previous similar message [ 415.711781] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.711984] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 415.750953] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 415.751764] Lustre: Skipped 1 previous similar message [ 416.470441] Lustre: server umount lustre-MDT0001 complete [ 420.312735] Lustre: server umount lustre-OST0000 complete [ 424.818629] Lustre: server umount lustre-OST0001 complete [ 433.670629] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 446.602590] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 494.564872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.919982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 494.994950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 495.156596] Lustre: lustre-MDT0000: new disk, initializing [ 495.260430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.311357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 500.357420] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 513.495498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.710196] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 513.776709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 513.781125] Lustre: Skipped 1 previous similar message [ 513.907255] Lustre: lustre-MDT0001: new disk, initializing [ 514.114857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 514.154411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 514.169576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 520.638525] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 533.078040] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 540.845843] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 549.675661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 550.132630] Lustre: lustre-OST0000: new disk, initializing [ 550.137461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 550.147730] Lustre: 17054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 550.267985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.233669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 551.255257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 551.444683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 557.760643] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 569.249044] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 574.929364] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 575.201569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 580.957870] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 581.255136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 587.233950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.256685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 587.262326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 587.263517] Lustre: Skipped 1 previous similar message [ 590.071701] Lustre: server umount lustre-OST0000 complete [ 595.941151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 595.953916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.983643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.996938] Lustre: Skipped 1 previous similar message [ 601.730645] Lustre: server umount lustre-MDT0000 complete [ 602.593690] LustreError: 17062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.630643] LustreError: 17062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 606.405258] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869627 with bad export cookie 13299436217861617057 [ 606.423280] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 606.424544] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.939803] Lustre: server umount lustre-MDT0001 complete [ 616.372682] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:13:54 (1788869634) [ 640.118248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.777730] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.875285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.116156] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.740281] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 650.211402] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.216662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.939771] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 672.001888] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 680.069654] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 682.969341] Lustre: Failing over lustre-MDT0000 [ 683.404717] Lustre: server umount lustre-MDT0000 complete [ 683.489153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 683.497401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 683.507811] Lustre: Skipped 1 previous similar message [ 683.511271] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 683.530657] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 702.945590] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.976931] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 703.984109] Lustre: 3649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788869708/real 1788869708] req@ffff9433c2db5880 x1875765285259264/t0(0) o400->MGC192.168.203.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788869724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 704.046217] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 705.436502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.184424] LustreError: 3645:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9433c384aa00 x1875765285261696/t0(0) o250->MGC192.168.203.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 713.565473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.571643] Lustre: Skipped 1 previous similar message [ 713.657872] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 718.832806] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 718.840524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 718.888848] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 719.780200] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 732.598960] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 738.287104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 738.309442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 738.326125] Lustre: Skipped 2 previous similar messages [ 738.336508] Lustre: Skipped 3 previous similar messages [ 743.337384] Lustre: server umount lustre-MDT0000 complete [ 743.402336] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 743.425985] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 748.029745] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869768 with bad export cookie 13299436217861618471 [ 748.031168] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 748.040733] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 748.685417] Lustre: server umount lustre-MDT0001 complete [ 761.366717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.980242] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 780.146822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.913374] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 795.367346] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 802.688833] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 805.720725] Lustre: Failing over lustre-MDT0000 [ 805.949839] Lustre: server umount lustre-MDT0000 complete [ 806.370181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 806.378155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 810.978141] LustreError: 23274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 811.013113] LustreError: 23274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 826.399483] Lustre: 3646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788869831/real 1788869831] req@ffff9433c384ad80 x1875765285290368/t0(0) o400->MGC192.168.203.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788869847 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 826.437834] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 827.897085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.575860] LustreError: 3645:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9433c384bb80 x1875765285293056/t0(0) o250->MGC192.168.203.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 837.025580] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 842.212126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 842.213513] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 842.217307] Lustre: Skipped 1 previous similar message [ 842.278039] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 842.682265] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 854.610286] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 860.641292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.644044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 860.659309] Lustre: Skipped 2 previous similar messages [ 860.678907] Lustre: Skipped 1 previous similar message [ 865.392088] Lustre: server umount lustre-MDT0000 complete [ 870.217965] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869891 with bad export cookie 13299436217861619409 [ 870.242203] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 870.243441] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.643832] Lustre: server umount lustre-MDT0001 complete [ 881.026857] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:18:19 (1788869899) [ 899.875626] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 911.373498] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 956.814362] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 969.583060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.837497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 969.865451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 969.962477] Lustre: lustre-MDT0000: new disk, initializing [ 970.075646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.078722] Lustre: Skipped 3 previous similar messages [ 970.100589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 975.004417] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 986.466442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.639364] Lustre: 29649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 987.006913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 992.392546] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1004.151373] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1010.848271] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1018.879300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1019.103764] Lustre: 31621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1020.569991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1020.629355] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1020.678476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1020.702515] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1025.651837] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1035.504227] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1041.590832] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1066.977777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1066.996233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1067.002072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1067.013375] Lustre: Skipped 1 previous similar message [ 1069.582861] Lustre: server umount lustre-OST0000 complete [ 1080.164889] Lustre: server umount lustre-MDT0000 complete [ 1082.337289] LustreError: 29663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.351388] LustreError: 29663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 1083.961969] LustreError: 29643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870104 with bad export cookie 13299436217861619815 [ 1083.975232] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1083.979592] LustreError: 29643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1084.481095] Lustre: server umount lustre-MDT0001 complete [ 1097.146719] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1100.460982] Key type lgssc unregistered [ 1100.797575] LNet: 33599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.810454] LNetError: 33599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.828756] LNet: Removed LNI 192.168.203.147@tcp [ 1101.892133] Key type .llcrypt unregistered [ 1101.896763] Key type ._llcrypt unregistered [ 1127.900950] Key type ._llcrypt registered [ 1127.903884] Key type .llcrypt registered [ 1128.030413] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1129.227475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1129.336425] alg: No test for adler32 (adler32-zlib) [ 1130.410611] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1130.658374] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1132.376133] Key type lgssc registered [ 1133.546672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1140.871927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1140.894879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1142.285092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.519828] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1152.482520] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.511476] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1154.511840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.806172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1158.713983] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1165.489589] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1171.284801] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1178.626619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1178.940848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.956579] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1184.948576] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1187.141756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1192.595854] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1209.178932] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:23:47 (1788870227) [ 1211.230245] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1213.147360] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1220.661733] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:23:59 (1788870239) [ 1231.830136] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:24:09 (1788870249) [ 1233.142765] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1244.077825] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1244.092863] Lustre: Skipped 2 previous similar messages [ 1246.632384] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:24:25 (1788870265) [ 1249.066425] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1249.074421] Lustre: Skipped 2 previous similar messages [ 1258.016547] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1258.027298] Lustre: Skipped 4 previous similar messages [ 1272.872108] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:24:51 (1788870291) [ 1275.683410] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1279.617201] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1290.773655] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:25:09 (1788870309) [ 1292.835774] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1292.840397] Lustre: Skipped 6 previous similar messages [ 1308.149343] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:25:26 (1788870326) [ 1323.490117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1323.504858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.538196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.541297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.549169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.550901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.566185] Lustre: Skipped 1 previous similar message [ 1327.612259] Lustre: server umount lustre-OST0000 complete [ 1334.240097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1334.258905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.281527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1337.826864] Lustre: server umount lustre-MDT0000 complete [ 1340.897428] LustreError: 35004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.918047] LustreError: 35004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1341.434585] LustreError: 37928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870362 with bad export cookie 9640463012175850836 [ 1341.442307] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1341.459258] LustreError: 37928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1341.862054] Lustre: server umount lustre-MDT0001 complete [ 1354.447663] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1357.427710] Key type lgssc unregistered [ 1357.681938] LNet: 42865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.686762] LNetError: 42865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.708918] LNet: Removed LNI 192.168.203.147@tcp [ 1358.636312] Key type .llcrypt unregistered [ 1358.638175] Key type ._llcrypt unregistered [ 1378.621155] Key type ._llcrypt registered [ 1378.629873] Key type .llcrypt registered [ 1378.791626] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 1397.061788] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1398.312521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1398.356253] alg: No test for adler32 (adler32-zlib) [ 1399.491863] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1399.864369] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1401.607151] Key type lgssc registered [ 1402.988991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1442.683705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1442.714550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.004294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1444.037398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1444.133622] Lustre: lustre-MDT0000: new disk, initializing [ 1444.229701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.244645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1449.034197] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1461.222300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.344937] Lustre: 46018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1461.404860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1461.414170] Lustre: Skipped 1 previous similar message [ 1461.491722] Lustre: lustre-MDT0001: new disk, initializing [ 1461.617952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1461.659066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1461.668407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1465.859024] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1475.997589] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1482.915932] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1490.680894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.056578] Lustre: lustre-OST0000: new disk, initializing [ 1491.060469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1491.066345] Lustre: 47986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1491.172508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.616110] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1500.252203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1500.267345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1500.362397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1508.832498] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1513.657605] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1513.881369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1519.135109] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1519.416427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1524.205556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.210921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1524.225730] Lustre: Skipped 1 previous similar message [ 1524.242262] Lustre: Skipped 1 previous similar message [ 1527.875839] Lustre: server umount lustre-OST0000 complete [ 1533.409765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1533.422488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.438192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1534.439307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1534.444747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.290728] Lustre: server umount lustre-MDT0000 complete [ 1541.798710] LustreError: 46009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870562 with bad export cookie 16979479381508188135 [ 1541.804373] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1541.809418] LustreError: 46009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1542.203070] Lustre: server umount lustre-MDT0001 complete [ 1552.606753] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:29:31 (1788870571) [ 1559.617513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.054783] LustreError: 50447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1560.078861] LustreError: 50447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1560.189856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1564.165605] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1565.667633] LustreError: 50448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1569.765660] LustreError: 50447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1571.971489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.335229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1576.824453] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1585.867656] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1593.025969] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1600.627361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1600.990417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.412770] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1614.263771] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1622.420495] LustreError: 52976:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1623.293893] Lustre: Modifying parameter general.timeout=1 in log params [ 1624.330310] Lustre: Modifying parameter general.timeout=2 in log params [ 1626.924448] Lustre: Modifying parameter general.timeout=5 in log params [ 1626.931533] Lustre: Skipped 2 previous similar messages [ 1630.949890] Lustre: Modifying parameter general.timeout=10 in log params [ 1630.953565] Lustre: Skipped 4 previous similar messages [ 1639.527305] Lustre: Modifying parameter general.timeout=19 in log params [ 1639.531508] Lustre: Skipped 8 previous similar messages [ 1655.587070] Lustre: Modifying parameter general.timeout=37 in log params [ 1655.590636] Lustre: Skipped 17 previous similar messages [ 1687.642819] Lustre: Modifying parameter general.timeout=74 in log params [ 1687.645358] Lustre: Skipped 36 previous similar messages [ 1747.499908] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:32:46 (1788870766) [ 1750.591259] Lustre: 59920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1752.588058] Lustre: 60020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1753.589081] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1753.594803] Lustre: Skipped 59 previous similar messages [ 1756.692396] Lustre: 60233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1756.708852] Lustre: 60233:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1786.858191] Lustre: 61783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1786.869593] Lustre: 61783:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1786.902646] LustreError: 61783:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1790.952947] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:33:30 (1788870810) [ 1804.256745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.259224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1804.282407] Lustre: Skipped 1 previous similar message [ 1804.286671] Lustre: Skipped 2 previous similar messages [ 1809.377591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1809.389961] Lustre: Skipped 4 previous similar messages [ 1810.506845] Lustre: server umount lustre-MDT0000 complete [ 1813.874752] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870834 with bad export cookie 16979479381508189017 [ 1813.880118] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1813.885351] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1814.150358] Lustre: server umount lustre-MDT0001 complete [ 1828.135212] Lustre: server umount lustre-OST0000 complete [ 1850.685539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.943763] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1851.236723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.482975] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1863.586189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.714537] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1863.748859] Lustre: 64316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1868.548688] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1875.304988] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1880.205625] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1887.392544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1887.602051] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1887.919173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.925655] Lustre: Skipped 1 previous similar message [ 1893.558378] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1900.538643] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1902.291215] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1902.302196] Lustre: Skipped 19 previous similar messages [ 1902.359900] Lustre: 66881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1911.779299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1911.803617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.826711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1916.238682] Lustre: server umount lustre-OST0000 complete [ 1920.481330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1920.491718] LustreError: Skipped 1 previous similar message [ 1920.497590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.509158] Lustre: Skipped 1 previous similar message [ 1932.769546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.776986] Lustre: Skipped 7 previous similar messages [ 1934.303273] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1934.505881] Lustre: server umount lustre-MDT0000 complete [ 1937.897633] LustreError: 64327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.940878] LustreError: 64327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1938.064646] LustreError: 64306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870958 with bad export cookie 16979479381508191992 [ 1938.071933] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1938.072938] LustreError: 64306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1938.481225] Lustre: server umount lustre-MDT0001 complete [ 1950.722172] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1954.028761] Key type lgssc unregistered [ 1954.480664] LNet: 68092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.492954] LNetError: 68092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.511935] LNet: Removed LNI 192.168.203.147@tcp [ 1955.596901] Key type .llcrypt unregistered [ 1955.602593] Key type ._llcrypt unregistered [ 1983.150878] Key type ._llcrypt registered [ 1983.153695] Key type .llcrypt registered [ 1983.244398] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1984.307610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1984.577653] alg: No test for adler32 (adler32-zlib) [ 1985.733591] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1985.980251] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1987.751384] Key type lgssc registered [ 1988.992380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1996.863160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1996.879716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.420814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.647465] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2008.549445] LustreError: 69502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.578098] LustreError: 69502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2012.355901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.052659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2017.492207] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2025.078122] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2032.354104] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2039.392032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2040.050652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2047.026075] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2055.990543] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2063.424912] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2073.058348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.061735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2073.089184] Lustre: Skipped 2 previous similar messages [ 2073.099721] Lustre: Skipped 2 previous similar messages [ 2078.178327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.190593] Lustre: Skipped 2 previous similar messages [ 2079.025840] Lustre: server umount lustre-MDT0000 complete [ 2082.865668] LustreError: 69482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871103 with bad export cookie 15869545212417653338 [ 2082.877677] LustreError: 69482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2082.879185] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2083.233950] Lustre: server umount lustre-MDT0001 complete [ 2096.579821] Lustre: server umount lustre-OST0000 complete [ 2125.484131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.810696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2125.842478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2126.178822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.722534] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2139.564591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.717366] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2139.787732] Lustre: 74371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2140.218996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2144.412952] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2151.608702] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2157.482580] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2164.583549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2164.827647] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2165.333412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.307607] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2180.715435] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2182.891519] Lustre: 76949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2184.897888] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2194.914993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2194.930406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.953996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2197.034529] Lustre: server umount lustre-OST0000 complete [ 2205.665747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.676188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2205.680730] Lustre: Skipped 2 previous similar messages [ 2205.696430] Lustre: Skipped 2 previous similar messages [ 2208.156536] Lustre: server umount lustre-MDT0000 complete [ 2210.790712] LustreError: 74381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.814141] LustreError: 74381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2212.074456] LustreError: 74366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871232 with bad export cookie 15869545212417654164 [ 2212.080618] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2212.091622] LustreError: 74366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2212.510166] Lustre: server umount lustre-MDT0001 complete [ 2224.657834] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2227.147270] Key type lgssc unregistered [ 2227.486736] LNet: 78158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.497066] LNetError: 78158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.583669] LNet: Removed LNI 192.168.203.147@tcp [ 2229.270823] Key type .llcrypt unregistered [ 2229.274780] Key type ._llcrypt unregistered [ 2257.697826] Key type ._llcrypt registered [ 2257.700335] Key type .llcrypt registered [ 2257.851374] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2258.667408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2258.730135] alg: No test for adler32 (adler32-zlib) [ 2260.089976] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 2260.350119] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2262.055149] Key type lgssc registered [ 2263.151692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2271.383476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2271.397430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.852858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2277.688343] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2282.978085] LustreError: 79566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.998227] LustreError: 79566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2286.900886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.557371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2292.016268] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2299.907891] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2306.997411] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2313.865959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2314.419761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.913393] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2328.237832] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2338.111361] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2339.133637] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2340.702969] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2340.705918] Lustre: Skipped 1 previous similar message [ 2343.386479] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2343.390163] Lustre: Skipped 2 previous similar messages [ 2348.004097] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2348.009545] Lustre: Skipped 4 previous similar messages [ 2356.254471] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2356.265304] Lustre: Skipped 8 previous similar messages [ 2372.494291] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2372.505815] Lustre: Skipped 18 previous similar messages [ 2405.037689] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2405.047306] Lustre: Skipped 34 previous similar messages [ 2436.004575] Lustre: 87668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2438.448609] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:44:17 (1788871457) [ 2439.920745] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2441.564584] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:44:20 (1788871460) [ 2445.796181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2445.796297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.808301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2445.812654] Lustre: Skipped 1 previous similar message [ 2445.822542] Lustre: Skipped 1 previous similar message [ 2450.701290] Lustre: server umount lustre-OST0000 complete [ 2456.032215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.034291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2456.050218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2456.054855] LustreError: Skipped 1 previous similar message [ 2456.065825] Lustre: Skipped 1 previous similar message [ 2459.931969] Lustre: server umount lustre-MDT0000 complete [ 2461.157804] LustreError: 79561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.171518] LustreError: 79561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2463.094561] LustreError: 79547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871483 with bad export cookie 1005210206085006349 [ 2463.104537] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2463.122718] LustreError: 79547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2463.405681] Lustre: server umount lustre-MDT0001 complete [ 2472.565363] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2474.725154] Key type lgssc unregistered [ 2474.937753] LNet: 88914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.949068] LNetError: 88914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.964956] LNet: Removed LNI 192.168.203.147@tcp [ 2475.724860] Key type .llcrypt unregistered [ 2475.727223] Key type ._llcrypt unregistered [ 2479.671457] Key type ._llcrypt registered [ 2479.675267] Key type .llcrypt registered [ 2479.818800] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2483.771561] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules [ 2484.767142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2484.823712] alg: No test for adler32 (adler32-zlib) [ 2486.007253] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 2486.250752] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2487.903183] Key type lgssc registered [ 2489.050633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2490.136879] LustreError: 90112:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2505.305544] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2512.012712] LustreError: 90112:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2513.761189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2513.789141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.241508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.258843] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2525.665573] LustreError: 91137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.690343] LustreError: 91137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2526.527613] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:45:45 (1788871545) [ 2529.914458] LustreError: 91168:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2529.925872] LustreError: 91168:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2529.936991] LustreError: 91168:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2530.211063] Lustre: server umount lustre-MDT0000 complete [ 2541.444934] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2543.890831] Key type lgssc unregistered [ 2544.236818] LNet: 92659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.245874] LNetError: 92659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.262927] LNet: Removed LNI 192.168.203.147@tcp [ 2545.061238] Key type .llcrypt unregistered [ 2545.065966] Key type ._llcrypt unregistered [ 2561.869933] Key type ._llcrypt registered [ 2561.872741] Key type .llcrypt registered [ 2561.968989] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2562.871280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2562.916405] alg: No test for adler32 (adler32-zlib) [ 2563.911353] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 2564.085147] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2565.743175] Key type lgssc registered [ 2566.632825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2572.987550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2573.012740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.507858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.215915] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2585.065242] LustreError: 93872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.073611] LustreError: 93872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2585.873630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.401661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2590.188423] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2596.630560] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2601.523784] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2607.305307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2607.835658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2613.339155] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2620.372383] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2665.528752] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:48:04 (1788871684) [ 2666.913379] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2668.589645] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:48:07 (1788871687) [ 2673.120542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2673.132934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.150696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.681032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.685489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.694776] Lustre: Skipped 1 previous similar message [ 2675.699815] Lustre: Skipped 1 previous similar message [ 2678.152577] Lustre: server umount lustre-MDT0000 complete [ 2680.802016] LustreError: 93873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.832516] LustreError: 93873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2681.906316] LustreError: 95724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871702 with bad export cookie 12813734063115421706 [ 2681.913095] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2681.916651] LustreError: 95724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2682.289477] Lustre: server umount lustre-MDT0001 complete [ 2696.530789] Lustre: server umount lustre-OST0000 complete [ 2706.193886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.497659] LustreError: 98149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.647558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.163427] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2712.034374] LustreError: 98150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.130601] LustreError: 98149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.758571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2717.257260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2720.574758] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2727.569352] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2732.358541] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2742.992301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2743.185354] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2743.197678] LustreError: 98136:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2743.221893] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2743.234876] LustreError: 100231:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2743.243278] LustreError: 100231:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2743.248872] LustreError: 100231:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2743.337162] Lustre: server umount lustre-OST0000 complete [ 2743.341412] LustreError: 100231:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2750.693350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2750.829654] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2750.836519] LustreError: 98136:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2750.847738] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2750.857428] LustreError: 100692:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2750.863835] LustreError: 100692:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2750.866809] LustreError: 100692:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2750.916705] Lustre: server umount lustre-OST0000 complete [ 2750.922278] LustreError: 100692:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2758.400033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2758.502368] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2758.504929] Lustre: Found index 0 for lustre-OST0000, updating log [ 2758.509410] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2758.556670] Lustre: lustre-OST0000: new disk, initializing [ 2758.559475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2758.564059] Lustre: 101225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2758.772804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2763.908657] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2767.203519] Lustre: Failing over lustre-OST0000 [ 2767.326353] Lustre: server umount lustre-OST0000 complete [ 2772.450640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.452884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2772.468787] Lustre: Skipped 1 previous similar message [ 2772.491848] Lustre: Skipped 2 previous similar messages [ 2776.662443] Lustre: server umount lustre-MDT0000 complete [ 2777.571096] LustreError: 98149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.592520] LustreError: 98149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2779.490736] LustreError: 98889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871800 with bad export cookie 12813734063115424443 [ 2779.498721] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2779.504327] LustreError: 98889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2789.977759] Lustre: server umount lustre-MDT0001 complete [ 2796.890385] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:50:16 (1788871816) [ 2809.632726] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2818.246440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.562733] LustreError: 103571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.653252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.615710] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2829.522735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2834.872752] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2837.445204] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2843.028698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.755749] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2848.743821] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2848.755143] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2848.776853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2848.943698] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2853.866575] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2853.880882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2854.889976] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 2854.922254] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2854.922932] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2856.178963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.250669] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2856.342336] Lustre: lustre-OST0001: new disk, initializing [ 2856.349705] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2856.358609] Lustre: 105889:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2856.776450] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2856.785498] Lustre: Skipped 2 previous similar messages [ 2862.977334] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2865.811949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2865.821930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2865.979851] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2870.523980] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2875.826962] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2880.950513] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.47@tcp (stopping) [ 2881.510798] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2881.511452] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.527532] LustreError: Skipped 1 previous similar message [ 2881.542408] Lustre: Skipped 1 previous similar message [ 2886.039133] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.47@tcp (stopping) [ 2886.050801] Lustre: Skipped 3 previous similar messages [ 2893.279218] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2893.422506] Lustre: server umount lustre-MDT0001 complete [ 2896.281798] LustreError: 103566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.295737] LustreError: 103566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2900.964958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.039982] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2901.043631] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2901.074740] Lustre: 103560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2902.121837] LustreError: 107429:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2902.133954] LustreError: 107429:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2902.140933] LustreError: 103562:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2902.247134] LustreError: 107439:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.147@tcp: cfg command failed: rc = -17 [ 2902.253752] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2902.253752] [ 2905.694670] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2906.604653] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2913.891044] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:52:13 (1788871933) [ 2928.126650] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2941.913647] Lustre: 109504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2960.197184] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2963.215970] Lustre: 110638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3044.831591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3044.833409] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.838048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3044.838058] Lustre: Skipped 7 previous similar messages [ 3044.863375] Lustre: Skipped 3 previous similar messages [ 3048.498287] Lustre: server umount lustre-MDT0000 complete [ 3049.954242] LustreError: 103565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.977867] LustreError: 103565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3051.824577] LustreError: 105088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872072 with bad export cookie 12813734063115425381 [ 3051.834052] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3051.840090] LustreError: 105088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3052.132839] Lustre: server umount lustre-MDT0001 complete [ 3065.238716] Lustre: server umount lustre-OST0000 complete [ 3078.337517] Lustre: server umount lustre-OST0001 complete [ 3083.542380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3093.066042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3102.026746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3111.706235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3132.865905] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3143.327910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3143.357083] Lustre: lustre-MDT0000: reset Object Index mappings [ 3144.111670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.122043] Lustre: Skipped 1 previous similar message [ 3148.861882] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3157.147350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.169313] Lustre: lustre-MDT0001: reset Object Index mappings [ 3162.397557] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3165.660542] Lustre: 115742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3171.550116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3179.004205] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3182.443417] LustreError: 116630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.449962] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 3182.459140] LustreError: 116630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3188.207492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3188.757454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3194.361695] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3194.370083] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3195.096435] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3203.104406] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3207.337954] Lustre: 117617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3259.258201] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:57:58 (1788872278) [ 3286.505142] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3286.507945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3286.518586] Lustre: Skipped 3 previous similar messages [ 3286.529838] Lustre: Skipped 6 previous similar messages [ 3291.965039] Lustre: server umount lustre-MDT0000 complete [ 3295.755570] LustreError: 114574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872316 with bad export cookie 12813734063115472253 [ 3295.765461] LustreError: 114574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3295.765758] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3302.265983] Lustre: server umount lustre-MDT0001 complete [ 3306.250491] LustreError: 118382:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x124:0x0].0x0 (ffff9433c3221300) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3312.559794] Lustre: server umount lustre-OST0000 complete [ 3323.039525] Lustre: server umount lustre-OST0001 complete [ 3330.433320] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 3341.766706] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3381.344984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3381.606825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3381.652328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3381.770521] Lustre: lustre-MDT0000: new disk, initializing [ 3381.833354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.840898] Lustre: Skipped 3 previous similar messages [ 3381.858247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3386.608759] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3393.937528] Lustre: server umount lustre-MDT0000 complete [ 3404.036100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.407227] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3404.436522] Lustre: 121895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3404.456496] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3409.112679] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3414.185063] Lustre: server umount lustre-MDT0000 complete [ 3423.296683] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:00:42 (1788872442) [ 3425.302422] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3427.423450] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:00:46 (1788872446) [ 3447.080992] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3458.449996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.241194] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3474.675073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.786618] Lustre: 123958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3474.830203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3474.833203] Lustre: Skipped 1 previous similar message [ 3474.903815] Lustre: lustre-MDT0001: new disk, initializing [ 3475.096796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3475.106267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3479.913579] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3485.557949] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3493.866079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3494.123550] Lustre: lustre-OST0000: new disk, initializing [ 3494.127947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3494.134740] Lustre: 125536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3496.099329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3496.108625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3496.277515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3502.160393] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3514.833478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.081792] Lustre: lustre-OST0001: new disk, initializing [ 3515.087746] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3515.101646] Lustre: 126407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3517.245548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3517.254952] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3517.442017] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3523.343493] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3535.155711] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3539.620449] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3963.481314] Lustre: 132827:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3963.490699] Lustre: 132827:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3963.495665] Lustre: 132827:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3963.507853] Lustre: 132827:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3963.522897] Lustre: 132827:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3963.534757] Lustre: 132827:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4272.937626] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:14:51 (1788873291) [ 4288.995944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4289.008724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.024584] Lustre: Skipped 3 previous similar messages [ 4289.028064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4289.036069] Lustre: Skipped 8 previous similar messages [ 4294.114089] Lustre: server umount lustre-MDT0000 complete [ 4295.653976] LustreError: 127209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.675752] LustreError: 127209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4297.916698] LustreError: 125535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873318 with bad export cookie 12813734063115541315 [ 4297.931050] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4300.772376] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4300.785418] Lustre: Skipped 3 previous similar messages [ 4300.793613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4300.801965] Lustre: Skipped 5 previous similar messages [ 4304.451190] Lustre: server umount lustre-MDT0001 complete [ 4314.369915] Lustre: server umount lustre-OST0000 complete [ 4324.422705] Lustre: server umount lustre-OST0001 complete [ 4331.981310] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 4342.371316] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4382.360837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4382.632409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4382.678721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4382.808516] Lustre: lustre-MDT0000: new disk, initializing [ 4382.930239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.943181] Lustre: Skipped 5 previous similar messages [ 4382.973879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4388.062774] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4400.026027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4400.139798] Lustre: 135887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4400.192613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4400.196210] Lustre: Skipped 1 previous similar message [ 4400.290463] Lustre: lustre-MDT0001: new disk, initializing [ 4400.539979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4400.564758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4405.781478] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4417.200364] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4425.218766] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4433.768606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4434.074382] Lustre: lustre-OST0000: new disk, initializing [ 4434.089306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4434.100633] Lustre: 137855:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4434.209535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4434.212665] Lustre: Skipped 1 previous similar message [ 4435.962341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4435.975675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4436.138326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4441.262202] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4451.440118] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4461.576933] Lustre: lustre-MDD0000: changelog on [ 4464.680765] Lustre: lustre-MDD0001: changelog on [ 6581.597449] Lustre: lustre-MDD0001: changelog off [ 6583.931457] Lustre: 142800:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6585.049321] Lustre: lustre-MDD0000: changelog off [ 6596.579697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6596.583435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.598436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6596.616626] Lustre: Skipped 2 previous similar messages [ 6596.642501] Lustre: Skipped 3 previous similar messages [ 6598.773479] Lustre: server umount lustre-OST0000 complete [ 6606.826685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.852110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6606.867768] Lustre: Skipped 1 previous similar message [ 6609.410607] Lustre: server umount lustre-MDT0000 complete [ 6611.940365] LustreError: 136735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6611.978732] LustreError: 136735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6613.528884] LustreError: 135878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875634 with bad export cookie 12813734063116363143 [ 6613.531187] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6613.552877] LustreError: 135878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6614.010940] Lustre: server umount lustre-MDT0001 complete [ 6626.193143] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 6629.603448] Key type lgssc unregistered [ 6629.957291] LNet: 147036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6629.964349] LNetError: 147036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6629.989406] LNet: Removed LNI 192.168.203.147@tcp [ 6631.101276] Key type .llcrypt unregistered [ 6631.104936] Key type ._llcrypt unregistered [ 6654.349741] Key type ._llcrypt registered [ 6654.352364] Key type .llcrypt registered [ 6654.444759] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 6671.436262] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 6672.172942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6672.324862] alg: No test for adler32 (adler32-zlib) [ 6673.470429] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 6673.735200] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 6675.407178] Key type lgssc registered [ 6676.494695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6712.921852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6712.943123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6714.174518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6714.212504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6714.302822] Lustre: lustre-MDT0000: new disk, initializing [ 6714.405461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6714.527392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6718.973668] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6729.428903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.632780] Lustre: 150236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6729.718078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6729.727887] Lustre: Skipped 1 previous similar message [ 6729.838614] Lustre: lustre-MDT0001: new disk, initializing [ 6729.924553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6729.955643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6729.967726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6734.384415] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6744.209791] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6751.500531] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6759.279682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6759.649419] Lustre: lustre-OST0000: new disk, initializing [ 6759.659795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6759.670739] Lustre: 152206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6759.806809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6766.474419] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6768.227501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6768.235640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6768.324711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6776.804325] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6782.810698] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_module obdecho/obdecho [ 6784.756794] LustreError: 153210:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6784.764798] LustreError: 153210:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6786.541865] LustreError: 153306:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6799.331830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6799.335214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6799.348664] Lustre: Skipped 2 previous similar messages [ 6799.359141] Lustre: Skipped 2 previous similar messages [ 6803.704155] Lustre: server umount lustre-MDT0000 complete [ 6807.663821] LustreError: 150230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875828 with bad export cookie 13954649782094550570 [ 6807.672737] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6807.989701] Lustre: server umount lustre-MDT0001 complete [ 6822.640868] Lustre: server umount lustre-OST0000 complete [ 6844.551815] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 6855.324350] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 6892.610888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6892.841210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6892.874655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6892.968917] Lustre: lustre-MDT0000: new disk, initializing [ 6893.075441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6893.089918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6897.559382] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6908.709647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.917193] Lustre: 157023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6909.016242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6909.023237] Lustre: Skipped 1 previous similar message [ 6909.133315] Lustre: lustre-MDT0001: new disk, initializing [ 6909.265805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6909.313663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6909.330859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6914.001293] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6923.489588] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6928.957911] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6935.458454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6935.721138] Lustre: lustre-OST0000: new disk, initializing [ 6935.729677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6935.735395] Lustre: 158993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6935.849181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.155917] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6942.795656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6942.806303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6942.897481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6951.725856] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6956.449566] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6956.686310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6961.790700] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6962.030147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6968.800957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6968.801575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6968.811901] LustreError: Skipped 1 previous similar message [ 6968.812027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6968.840385] Lustre: Skipped 1 previous similar message [ 6970.154272] Lustre: server umount lustre-OST0000 complete [ 6975.968943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6975.975573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.998388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6976.006480] Lustre: Skipped 1 previous similar message [ 6979.042948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6979.046587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6980.671583] Lustre: server umount lustre-MDT0000 complete [ 6984.171188] LustreError: 157027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.190800] LustreError: 157027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6984.529178] LustreError: 158992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876005 with bad export cookie 13954649782094551529 [ 6984.538261] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6984.544793] LustreError: 158992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6985.043212] Lustre: server umount lustre-MDT0001 complete [ 6988.955610] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:00:07 (1788876007) [ 6990.919712] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6993.246549] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:00:12 (1788876012) [ 7001.937749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7002.618107] LustreError: 161404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7002.700183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7007.648330] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7007.723473] LustreError: 161405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7012.834152] LustreError: 161404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7016.864388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.266788] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7031.186901] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7037.885940] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7045.635800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7046.045695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7046.050409] Lustre: Skipped 1 previous similar message [ 7053.108044] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7061.322411] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7072.743983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7072.746364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7072.767756] Lustre: Skipped 2 previous similar messages [ 7072.782860] Lustre: Skipped 3 previous similar messages [ 7078.951936] Lustre: server umount lustre-MDT0000 complete [ 7082.400201] LustreError: 163368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7083.171776] LustreError: 161386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876104 with bad export cookie 13954649782094552418 [ 7083.173769] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7083.178631] LustreError: 161386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7087.548747] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.47@tcp (stopping) [ 7087.556986] Lustre: Skipped 4 previous similar messages [ 7088.106250] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7089.697510] Lustre: server umount lustre-MDT0001 complete [ 7096.767644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7112.706618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7112.935398] LustreError: 165260:0:(llog.c:1655:llog_backup()) MGC192.168.203.147@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7112.941435] Lustre: 165260:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.147@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7118.229751] LustreError: 165265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7118.255954] LustreError: 165265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7135.211687] LustreError: 165266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.233216] LustreError: 165266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 7168.409681] LustreError: 165265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7168.427538] LustreError: 165265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 7173.601414] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7173.613915] Lustre: 165260:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.147@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7214.601372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7219.579691] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7219.685343] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7229.545791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7229.931267] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc1a8e00ababa7698 [ 7229.942843] Lustre: MGC192.168.203.147@tcp: Connection restored to 0@lo (at 0@lo) [ 7235.570109] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7236.015593] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7245.147509] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7247.021980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7253.926336] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7255.946400] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7276.517746] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7276.520477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7276.522274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7276.538889] Lustre: Skipped 1 previous similar message [ 7276.564630] Lustre: Skipped 3 previous similar messages [ 7282.898575] Lustre: server umount lustre-MDT0000 complete [ 7286.705116] LustreError: 161385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876307 with bad export cookie 13954649782094558872 [ 7286.711390] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7286.721663] LustreError: 161385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7286.753160] LustreError: 165283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7286.771246] LustreError: 165283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7287.035465] Lustre: server umount lustre-MDT0001 complete [ 7293.709819] Lustre: server umount lustre-OST0000 complete [ 7318.641674] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 7331.590226] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7375.382160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7375.733691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7375.796568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7375.960874] Lustre: lustre-MDT0000: new disk, initializing [ 7376.051696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7376.057738] Lustre: Skipped 1 previous similar message [ 7376.073824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7381.658699] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7393.721328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.869327] Lustre: 170370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7394.178680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7399.615064] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7410.451695] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7418.314486] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7426.298816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7426.496294] Lustre: lustre-OST0000: new disk, initializing [ 7426.501713] Lustre: Skipped 1 previous similar message [ 7426.507418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7426.521749] Lustre: Skipped 2 previous similar messages [ 7426.526293] Lustre: 172340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7432.974563] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7436.399387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7436.414486] Lustre: Skipped 1 previous similar message [ 7436.429519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7436.495587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7443.807794] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7449.834501] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7450.188435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7455.967923] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7456.302173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7462.368125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7462.376502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7462.381379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7462.407346] Lustre: Skipped 3 previous similar messages [ 7462.416317] Lustre: Skipped 4 previous similar messages [ 7464.927951] Lustre: server umount lustre-OST0000 complete [ 7471.072196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7471.076331] LustreError: Skipped 1 previous similar message [ 7483.360041] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7483.647396] Lustre: server umount lustre-MDT0000 complete [ 7487.688392] LustreError: 170361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876508 with bad export cookie 13954649782094575707 [ 7487.698672] LustreError: 170361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7487.700368] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7487.950507] Lustre: server umount lustre-MDT0001 complete [ 7498.745772] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:08:37 (1788876517) [ 7506.391506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.718462] LustreError: 174753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7512.279697] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7521.153293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7526.283572] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7534.983892] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7541.540520] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7548.440388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7555.461434] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7563.707767] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7577.902313] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:09:56 (1788876596) [ 7582.689475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7582.701439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7582.718724] Lustre: Skipped 2 previous similar messages [ 7582.725659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7582.729873] Lustre: Skipped 8 previous similar messages [ 7588.894221] Lustre: server umount lustre-OST0000 complete [ 7598.923446] Lustre: server umount lustre-MDT0000 complete [ 7602.903957] LustreError: 174733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876623 with bad export cookie 13954649782094576589 [ 7602.909337] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7602.911924] LustreError: 174733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7603.242987] Lustre: server umount lustre-MDT0001 complete [ 7615.146172] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7618.269595] Key type lgssc unregistered [ 7618.547550] LNet: 178480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7618.552962] LNetError: 178480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7618.573482] LNet: Removed LNI 192.168.203.147@tcp [ 7619.542354] Key type .llcrypt unregistered [ 7619.550420] Key type ._llcrypt unregistered [ 7639.522835] Key type ._llcrypt registered [ 7639.526158] Key type .llcrypt registered [ 7639.635120] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7640.842883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7641.037140] alg: No test for adler32 (adler32-zlib) [ 7642.138652] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 7642.329977] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7643.991564] Key type lgssc registered [ 7645.539964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7653.906638] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7653.958218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7669.663374] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7674.784124] LustreError: 179720:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 7679.904629] LustreError: 179720:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7679.922385] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7679.937624] LustreError: MGC192.168.203.147@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7679.951300] LustreError: 179720:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7679.961385] LustreError: 179720:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7679.973644] LustreError: 179720:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7680.050571] Lustre: server umount lustre-OST0000 complete [ 7680.054846] LustreError: 179720:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7689.790237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7691.278973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7695.974823] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7701.478466] LustreError: 180220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7701.514930] LustreError: 180220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7705.971450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.371749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7711.536978] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7719.835198] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7727.097920] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7735.114233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7735.316027] LustreError: 182037:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7735.323317] Lustre: 182037:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.147@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7735.451278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7742.926684] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7751.398424] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7757.291237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7757.292840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7757.324015] Lustre: Skipped 1 previous similar message [ 7760.501654] Lustre: server umount lustre-MDT0000 complete [ 7762.416183] LustreError: 182798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7762.447817] LustreError: 182798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7764.125249] LustreError: 180201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876785 with bad export cookie 14888771479907463306 [ 7764.127016] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7764.136947] LustreError: 180201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7764.538703] Lustre: server umount lustre-MDT0001 complete [ 7778.461344] Lustre: server umount lustre-OST0000 complete [ 7790.361390] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 7802.519825] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7847.101455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7847.399236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7847.426988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7847.547348] Lustre: lustre-MDT0000: new disk, initializing [ 7847.662824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7847.711615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7853.733593] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7867.204364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7867.384411] Lustre: 185568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7867.429101] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7867.436088] Lustre: Skipped 1 previous similar message [ 7867.549409] Lustre: lustre-MDT0001: new disk, initializing [ 7867.631384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7867.669130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7867.686647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7872.834379] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7883.806399] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7890.727925] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7898.205045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7898.454775] Lustre: lustre-OST0000: new disk, initializing [ 7898.457829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7898.464462] Lustre: 187536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7898.569818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7905.424621] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7907.408339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7907.423457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7907.475358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7916.996814] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7922.345650] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7922.564067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7927.966302] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7928.195310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7933.416172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7933.426888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.437364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7933.447823] Lustre: Skipped 2 previous similar messages [ 7933.467067] Lustre: Skipped 3 previous similar messages [ 7936.768348] Lustre: server umount lustre-OST0000 complete [ 7943.648385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.665747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7948.084257] Lustre: server umount lustre-MDT0000 complete [ 7948.774949] LustreError: 185574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.802476] LustreError: 185574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7952.872186] LustreError: 186370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876973 with bad export cookie 14888771479907464027 [ 7952.879474] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7952.883226] LustreError: 186370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7953.435709] Lustre: server umount lustre-MDT0001 complete [ 7965.193672] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:16:23 (1788876983) [ 7986.645985] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7990.334464] Key type lgssc unregistered [ 7990.824159] LNet: 190482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7990.836996] LNetError: 190482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7990.860261] LNet: Removed LNI 192.168.203.147@tcp [ 7991.993742] Key type .llcrypt unregistered [ 7991.998491] Key type ._llcrypt unregistered [ 8012.983880] Key type ._llcrypt registered [ 8012.986891] Key type .llcrypt registered [ 8013.130444] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8013.940919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8014.036447] alg: No test for adler32 (adler32-zlib) [ 8015.510740] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 8015.815251] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8017.591260] Key type lgssc registered [ 8019.115547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8029.413932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8029.447251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8031.124317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8036.835450] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8040.425885] LustreError: 191708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8040.446114] LustreError: 191708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8045.536565] LustreError: 191707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8046.804651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8047.149268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8050.817042] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8056.288757] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8060.191289] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8064.993498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8065.101861] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8065.103512] LustreError: 193571:0:(llog.c:1696:llog_backup()) MGC192.168.203.147@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8065.107526] Lustre: 193571:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.147@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8065.182147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.690633] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8072.603715] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8076.770897] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8076.774311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8076.776875] Lustre: Skipped 1 previous similar message [ 8076.784598] Lustre: Skipped 2 previous similar messages [ 8081.890154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8081.899099] Lustre: Skipped 2 previous similar messages [ 8087.012483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8087.017484] Lustre: Skipped 2 previous similar messages [ 8088.032453] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8088.137945] Lustre: server umount lustre-MDT0000 complete [ 8090.097148] LustreError: 191687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877111 with bad export cookie 5622000762321669452 [ 8090.106444] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8090.289094] Lustre: server umount lustre-MDT0001 complete [ 8102.562263] Lustre: server umount lustre-OST0000 complete [ 8106.246285] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8110.277576] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8124.804065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.902100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8124.913617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8124.955079] Lustre: lustre-MDT0000: new disk, initializing [ 8124.983269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8124.990056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8126.637169] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8131.738473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.794084] Lustre: 197091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8131.813205] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8131.815166] Lustre: Skipped 1 previous similar message [ 8131.816904] LustreError: 197874:0:(llog.c:1696:llog_backup()) MGC192.168.203.147@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8131.820381] LustreError: 197874:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8131.823938] Lustre: 197874:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.147@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8131.832371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8131.834313] Lustre: Skipped 1 previous similar message [ 8131.874743] Lustre: lustre-MDT0001: new disk, initializing [ 8131.912024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8131.924197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8131.927403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8133.473707] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8137.681387] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8139.847565] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8142.462784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8142.582604] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8142.584501] Lustre: Skipped 1 previous similar message [ 8142.585954] LustreError: 199035:0:(llog.c:1696:llog_backup()) MGC192.168.203.147@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8142.589568] LustreError: 199035:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8142.592517] Lustre: 199035:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.147@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8142.596967] Lustre: 199035:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8142.607702] Lustre: lustre-OST0000: new disk, initializing [ 8142.610658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8142.614087] Lustre: 199061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8142.644996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8144.451281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8144.456122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8144.504943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8144.866531] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8148.780554] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8150.364492] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8150.434699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8151.888460] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8151.960996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8154.593310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8154.594271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.594982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8154.600887] Lustre: Skipped 2 previous similar messages [ 8158.731962] Lustre: server umount lustre-OST0000 complete [ 8162.783716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8162.786647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8162.790378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8162.791949] Lustre: Skipped 1 previous similar message [ 8164.832419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8166.107288] Lustre: server umount lustre-MDT0000 complete [ 8167.278075] LustreError: 199060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877188 with bad export cookie 5622000762321670166 [ 8167.281257] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8167.282296] LustreError: 199060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8167.393342] Lustre: server umount lustre-MDT0001 complete [ 8170.230114] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:19:50 (1788877190) [ 8175.712103] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8179.101914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8179.248761] LustreError: 202180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8179.258535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.598685] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8183.317767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.744903] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8185.647930] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8187.649252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8189.597860] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8192.450719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.512229] Lustre: lustre-OST0001: new disk, initializing [ 8192.514459] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8192.517801] Lustre: 204485:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8194.500887] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8195.585351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8195.589712] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8195.632212] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8199.837920] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8205.546370] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8206.136793] Lustre: DEBUG MARKER: ADD OST3 [ 8207.773847] loop: module loaded [ 8207.790982] loop0: detected capacity change from 0 to 204800000 [ 8207.909901] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8207.915312] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8207.919542] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8209.876397] loop0: detected capacity change from 0 to 204800000 [ 8211.613718] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8211.672937] Lustre: lustre-OST0003: new disk, initializing [ 8211.674482] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8211.676788] Lustre: 206217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8211.715415] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8211.717232] Lustre: Skipped 3 previous similar messages [ 8212.841723] Lustre: DEBUG MARKER: STOP OST3 [ 8213.439909] Lustre: Failing over lustre-OST0003 [ 8213.479282] Lustre: server umount lustre-OST0003 complete [ 8213.613515] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8213.636802] LustreError: 203663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8217.582236] Lustre: DEBUG MARKER: START OST3 again [ 8218.014509] LustreError: 204057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8218.018659] LustreError: 204057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8219.672901] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8219.765044] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8221.463902] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8221.471707] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 8221.540951] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8225.264481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8225.267220] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8225.275167] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 8235.005624] Lustre: Failing over lustre-OST0003 [ 8235.052656] Lustre: server umount lustre-OST0003 complete [ 8235.494159] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8235.494376] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8235.496938] LustreError: Skipped 1 previous similar message [ 8235.503755] LustreError: 203665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8235.511735] LustreError: 203665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8250.335860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8250.337898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.341348] Lustre: Skipped 1 previous similar message [ 8250.343548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8250.345329] Lustre: Skipped 2 previous similar messages [ 8253.797933] Lustre: server umount lustre-MDT0000 complete [ 8255.054610] LustreError: 202162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877275 with bad export cookie 5622000762321671034 [ 8255.059405] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8255.060348] LustreError: 202162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8255.968204] LustreError: 202176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8255.975705] LustreError: 202176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8261.367534] Lustre: server umount lustre-MDT0001 complete [ 8262.656267] Lustre: server umount lustre-OST0000 complete [ 8263.910958] Lustre: server umount lustre-OST0001 complete [ 8265.956124] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8268.438026] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8281.134978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8281.237279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8281.246887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8281.287253] Lustre: lustre-MDT0000: new disk, initializing [ 8281.308717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8281.310945] Lustre: Skipped 1 previous similar message [ 8281.316062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8282.703406] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8286.826580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8286.871688] Lustre: 211627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8287.011568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8288.335437] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8292.094812] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8293.982054] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8296.382678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8296.505205] Lustre: 213596:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8298.434772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8298.439053] Lustre: Skipped 1 previous similar message [ 8298.442296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8298.488439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8298.584659] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8302.343671] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8303.698980] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8303.768339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8305.101112] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8305.176555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8307.862111] Lustre: server umount lustre-OST0000 complete [ 8312.799980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8312.803483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8312.808921] Lustre: Skipped 5 previous similar messages [ 8312.811428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8312.814226] Lustre: Skipped 7 previous similar messages [ 8319.995878] Lustre: server umount lustre-MDT0000 complete [ 8321.228278] LustreError: 211619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877342 with bad export cookie 5622000762321673274 [ 8321.231775] LustreError: 211619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8321.232277] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8331.823511] Lustre: server umount lustre-MDT0001 complete [ 8333.081725] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:22:33 (1788877353) [ 8338.739382] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8342.248062] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8355.602090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8355.693031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8355.703963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8355.705734] Lustre: Skipped 3 previous similar messages [ 8355.743632] Lustre: lustre-MDT0000: new disk, initializing [ 8355.744982] Lustre: Skipped 2 previous similar messages [ 8355.769106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8357.350704] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8361.673962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.721138] Lustre: 218199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8361.835276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8363.398270] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8367.538751] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8369.553655] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8372.515370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8372.629449] Lustre: 220165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8373.732944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8374.663128] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8378.321342] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8379.871195] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8379.936198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8381.395340] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8381.509088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8383.969760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8383.971154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8383.974669] LustreError: Skipped 1 previous similar message [ 8383.980717] Lustre: Skipped 1 previous similar message [ 8383.981928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8383.988252] Lustre: Skipped 5 previous similar messages [ 8384.336726] Lustre: server umount lustre-OST0000 complete [ 8391.786864] Lustre: server umount lustre-MDT0000 complete [ 8393.125120] LustreError: 218234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877414 with bad export cookie 5622000762321674156 [ 8393.128112] LustreError: 218234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8393.128871] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8393.249668] Lustre: server umount lustre-MDT0001 complete [ 8396.043676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8396.196913] LustreError: 222376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8396.204056] LustreError: 222376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8397.563635] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8400.318806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8401.885761] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8404.606098] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8406.776414] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8409.572224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8409.719724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.722478] Lustre: Skipped 7 previous similar messages [ 8411.957960] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8414.749128] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8431.583887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8431.587263] LustreError: Skipped 1 previous similar message [ 8437.651863] Lustre: server umount lustre-OST0000 complete [ 8445.085823] Lustre: server umount lustre-MDT0000 complete [ 8446.318595] LustreError: 222357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877467 with bad export cookie 5622000762321675038 [ 8446.323076] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8446.324955] LustreError: 222357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8446.456745] Lustre: server umount lustre-MDT0001 complete [ 8547.731117] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8548.965640] Key type lgssc unregistered [ 8549.096570] LNet: 226054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8549.100863] LNetError: 226054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8549.112432] LNet: Removed LNI 192.168.203.147@tcp [ 8549.503106] Key type .llcrypt unregistered [ 8549.504211] Key type ._llcrypt unregistered [ 8558.647335] Key type ._llcrypt registered [ 8558.648555] Key type .llcrypt registered [ 8558.691335] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8565.612405] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8566.003085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8566.024968] alg: No test for adler32 (adler32-zlib) [ 8566.897629] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 8566.984581] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8568.559140] Key type lgssc registered [ 8568.943098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8581.711558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8581.720437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8582.832455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8582.843808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8582.885841] Lustre: lustre-MDT0000: new disk, initializing [ 8582.911720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8582.918965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8584.492621] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8589.309914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8589.374315] Lustre: 229374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8589.414763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8589.416970] Lustre: Skipped 1 previous similar message [ 8589.460882] Lustre: lustre-MDT0001: new disk, initializing [ 8589.502295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8589.529635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8589.535282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8591.042386] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8595.078491] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8596.854875] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8599.515316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8599.638186] Lustre: lustre-OST0000: new disk, initializing [ 8599.640286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8599.643168] Lustre: 231343:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8599.678141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8601.826562] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8602.385204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8602.391009] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8602.437535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8606.209863] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8608.004266] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8608.131451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8610.102939] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8610.207521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8612.840387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8612.840755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8612.845987] LustreError: Skipped 1 previous similar message [ 8612.847159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8612.855940] Lustre: Skipped 1 previous similar message [ 8617.956223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8617.960442] Lustre: Skipped 2 previous similar messages [ 8623.078070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8623.082870] Lustre: Skipped 1 previous similar message [ 8625.119091] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8625.169478] Lustre: server umount lustre-OST0000 complete [ 8628.194853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.197218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8628.202952] Lustre: Skipped 1 previous similar message [ 8632.453918] Lustre: server umount lustre-MDT0000 complete [ 8634.083399] LustreError: 229367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877654 with bad export cookie 8914156483793804765 [ 8634.086394] LustreError: 229367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8634.086455] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8634.206641] Lustre: server umount lustre-MDT0001 complete [ 8640.928786] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8644.814865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8644.987549] LustreError: 234273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8644.997094] LustreError: 234273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8645.005316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.399430] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8649.478958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8649.630056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8651.214928] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8652.449174] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8654.729502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8656.671511] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8659.639177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8659.707291] Lustre: lustre-OST0001: new disk, initializing [ 8659.709162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8659.711758] Lustre: 236577:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8659.742724] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8659.744582] Lustre: Skipped 1 previous similar message [ 8661.766064] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8662.594875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8662.598556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8662.630787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8667.347388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8674.140752] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8678.576111] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:28:18 (1788877698) [ 8680.420680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8680.420944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8680.429703] Lustre: Skipped 3 previous similar messages [ 8680.431362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8680.440723] Lustre: Skipped 4 previous similar messages [ 8685.662246] Lustre: server umount lustre-MDT0000 complete [ 8686.845667] LustreError: 235406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877707 with bad export cookie 8914156483793805647 [ 8686.851211] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8686.970756] Lustre: server umount lustre-MDT0001 complete [ 8698.886071] Lustre: server umount lustre-OST0000 complete [ 8710.588428] Lustre: server umount lustre-OST0001 complete [ 8713.215801] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8717.248628] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8731.570223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8731.674189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8731.692819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8731.737212] Lustre: lustre-MDT0000: new disk, initializing [ 8731.759639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8731.767174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8733.314541] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8738.047173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.100280] Lustre: 240518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8738.236013] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8739.885261] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8744.601027] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8747.028698] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8749.922581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8750.123727] Lustre: lustre-OST0000: new disk, initializing [ 8750.126303] Lustre: Skipped 1 previous similar message [ 8750.129111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8750.132222] Lustre: Skipped 2 previous similar messages [ 8750.135512] Lustre: 242485:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8751.954344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8751.959566] Lustre: Skipped 1 previous similar message [ 8751.963900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8752.020371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8752.932674] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8757.545899] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8759.641371] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8759.758302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8761.576730] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8761.665402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8767.460575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8767.461041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.465568] LustreError: Skipped 1 previous similar message [ 8767.466593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8767.473700] Lustre: Skipped 1 previous similar message [ 8767.476473] Lustre: Skipped 4 previous similar messages [ 8768.609554] Lustre: server umount lustre-OST0000 complete [ 8772.579838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8772.588792] Lustre: Skipped 1 previous similar message [ 8776.479898] Lustre: server umount lustre-MDT0000 complete [ 8777.696847] LustreError: 240530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8777.701675] LustreError: 240530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8777.889170] LustreError: 240512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877798 with bad export cookie 8914156483793807740 [ 8777.892553] LustreError: 240512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8777.896352] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8778.051884] Lustre: server umount lustre-MDT0001 complete [ 8782.257151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8782.467995] LustreError: 244699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8782.501940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8782.504876] Lustre: Skipped 2 previous similar messages [ 8784.212227] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8787.145965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8788.968108] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8792.590255] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8795.098426] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8797.734957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8800.289406] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8803.518756] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8950.753122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8950.755087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8950.755390] LustreError: Skipped 1 previous similar message [ 8950.759205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8950.761765] Lustre: Skipped 4 previous similar messages [ 8952.013270] Lustre: server umount lustre-OST0000 complete [ 8959.382860] Lustre: server umount lustre-MDT0000 complete [ 8960.561198] LustreError: 245433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877981 with bad export cookie 8914156483793808622 [ 8960.564476] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8960.564969] LustreError: 245433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8960.684958] Lustre: server umount lustre-MDT0001 complete [ 9072.359889] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9073.399755] Key type lgssc unregistered [ 9073.531510] LNet: 248306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9073.534823] LNetError: 248306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9073.541312] LNet: Removed LNI 192.168.203.147@tcp [ 9073.884156] Key type .llcrypt unregistered [ 9073.885834] Key type ._llcrypt unregistered [ 9082.867965] Key type ._llcrypt registered [ 9082.869245] Key type .llcrypt registered [ 9082.907603] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 9089.868932] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9090.189545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9090.197433] alg: No test for adler32 (adler32-zlib) [ 9091.055831] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 9091.146374] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9092.727182] Key type lgssc registered [ 9093.205058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9109.931652] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9113.703314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9113.709966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9114.795606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9114.806115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9114.842362] Lustre: lustre-MDT0000: new disk, initializing [ 9114.861989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9114.867577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9116.219543] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9120.561856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9120.599618] Lustre: 252339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9120.634733] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9120.635515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9120.639197] Lustre: Skipped 1 previous similar message [ 9120.673683] Lustre: lustre-MDT0001: new disk, initializing [ 9120.704073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9120.716130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9120.720073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9122.068614] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9124.402997] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9126.846836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9126.961668] Lustre: lustre-OST0000: new disk, initializing [ 9126.963726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9126.967250] Lustre: 253973:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9127.040822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9128.955541] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9132.522391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9132.525543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9132.538026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9133.167963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9133.278322] Lustre: lustre-OST0001: new disk, initializing [ 9133.280414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9133.283223] Lustre: 254847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9133.342467] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9135.693372] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9135.966720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9135.971199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9136.010176] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9141.503892] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9144.207581] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9151.460323] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.463031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9151.463978] Lustre: Skipped 3 previous similar messages [ 9151.464073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9153.451418] Lustre: server umount lustre-MDT0000 complete [ 9154.738692] LustreError: 252332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878175 with bad export cookie 16055559367454834153 [ 9154.742400] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9154.744040] LustreError: 252332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9154.874372] Lustre: server umount lustre-MDT0001 complete [ 9166.791634] Lustre: server umount lustre-OST0000 complete [ 9178.524249] Lustre: server umount lustre-OST0001 complete [ 9183.053846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9186.033675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9188.625491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9191.723865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9198.127021] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9201.818510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9201.825649] Lustre: lustre-MDT0000: reset Object Index mappings [ 9201.986336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9203.390917] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9206.374285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9206.381279] Lustre: lustre-MDT0001: reset Object Index mappings [ 9207.925252] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9208.964308] Lustre: 260985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9211.787354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9211.924966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9211.927551] Lustre: Skipped 1 previous similar message [ 9213.810308] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9216.950558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9219.039968] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9226.231828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9226.233683] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9233.345718] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9235.035387] Lustre: 262876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9237.473830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9237.479083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9237.488497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9237.493989] Lustre: Skipped 3 previous similar messages [ 9241.570347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9241.573357] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9241.579413] Lustre: Skipped 1 previous similar message [ 9242.750173] Lustre: server umount lustre-MDT0000 complete [ 9244.457829] LustreError: 259828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878265 with bad export cookie 16055559367454840915 [ 9244.463558] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9244.464887] LustreError: 259828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9244.600153] Lustre: server umount lustre-MDT0001 complete [ 9256.900731] Lustre: server umount lustre-OST0000 complete [ 9267.299705] Lustre: server umount lustre-OST0001 complete [ 9274.309463] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:38:14 (1788878294) [ 9280.381326] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 9283.826462] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9296.338911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9296.422354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9296.430927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9296.464467] Lustre: lustre-MDT0000: new disk, initializing [ 9296.482968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9296.484850] Lustre: Skipped 1 previous similar message [ 9296.489184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9297.757572] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9302.401337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9302.446084] Lustre: 267120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9302.557045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9304.271265] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9308.554732] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9310.718807] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9313.307712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9313.404625] Lustre: lustre-OST0000: new disk, initializing [ 9313.406955] Lustre: Skipped 1 previous similar message [ 9313.409310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9313.412276] Lustre: Skipped 2 previous similar messages [ 9313.415063] Lustre: 269089:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9314.810882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9314.813564] Lustre: Skipped 1 previous similar message [ 9314.815553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9314.849150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9315.439693] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9319.179137] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9320.719120] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9320.814085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9322.352250] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9322.448685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9325.031975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9325.033076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9325.037749] LustreError: Skipped 1 previous similar message [ 9325.049796] Lustre: Skipped 1 previous similar message [ 9325.051069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9325.061509] Lustre: Skipped 4 previous similar messages [ 9329.232246] Lustre: server umount lustre-OST0000 complete [ 9333.216674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9333.224918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9333.237563] Lustre: Skipped 1 previous similar message [ 9333.243069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9336.644390] Lustre: server umount lustre-MDT0000 complete [ 9338.473642] LustreError: 267115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878359 with bad export cookie 16055559367454842987 [ 9338.477600] LustreError: 267115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9338.477801] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9338.617907] Lustre: server umount lustre-MDT0001 complete [ 9347.482228] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9352.395253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9352.594075] LustreError: 272018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9352.602942] LustreError: 272018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9352.612688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9352.615839] Lustre: Skipped 2 previous similar messages [ 9354.270824] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9357.794608] LustreError: 272019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9358.189069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9360.157830] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9361.547953] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9364.235909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9367.365640] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9371.164292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9371.292686] Lustre: lustre-OST0001: new disk, initializing [ 9371.295535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9371.299903] Lustre: 274325:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9372.443254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9372.448384] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9372.502916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9373.875474] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9379.082464] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9380.437633] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9382.636370] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9382.640687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9388.004521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9388.009266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9388.010963] Lustre: Skipped 3 previous similar messages [ 9388.013982] Lustre: Skipped 3 previous similar messages [ 9390.869869] Lustre: server umount lustre-MDT0000 complete [ 9392.483494] LustreError: 271998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878413 with bad export cookie 16055559367454843869 [ 9392.488671] LustreError: 271998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9392.489912] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9392.611707] Lustre: server umount lustre-MDT0001 complete [ 9404.367388] Lustre: server umount lustre-OST0000 complete [ 9415.223380] Lustre: server umount lustre-OST0001 complete [ 9421.918787] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9426.417645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9426.561165] LustreError: 277376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9426.585720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9426.587807] Lustre: Skipped 3 previous similar messages [ 9428.489932] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9432.034947] LustreError: 277377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9432.088597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9434.116251] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9435.469859] Lustre: 278519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9438.161863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9440.454745] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9442.405604] LustreError: 278873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.673696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9446.692970] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9452.027630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9452.044659] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9455.683245] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9457.099168] Lustre: 280395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9459.129277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9459.135116] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9462.642241] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:41:22 (1788878482) [ 9477.604426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9477.612217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9477.616688] Lustre: Skipped 3 previous similar messages [ 9477.621433] Lustre: Skipped 3 previous similar messages [ 9480.531454] Lustre: server umount lustre-MDT0000 complete [ 9482.224355] LustreError: 277357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878503 with bad export cookie 16055559367454850148 [ 9482.228242] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9482.229099] LustreError: 277357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9482.720320] LustreError: 280152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9482.728213] LustreError: 280152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9488.476230] Lustre: server umount lustre-MDT0001 complete [ 9504.223110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9504.282378] Lustre: server umount lustre-OST0000 complete [ 9520.095140] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9520.197740] Lustre: server umount lustre-OST0001 complete [ 9522.975060] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 9527.653316] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9547.171966] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9550.650418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9550.733579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9550.742629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9550.780548] Lustre: lustre-MDT0000: new disk, initializing [ 9550.804409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9552.115298] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9556.719144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.770325] Lustre: 284629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9556.871458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9556.873226] Lustre: Skipped 4 previous similar messages [ 9556.883826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9558.215723] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9563.272243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9563.405995] Lustre: 286262:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9565.041752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9565.612201] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9570.619382] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9572.327052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9572.777358] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9577.588274] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9583.919154] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:43:24 (1788878604) [ 9587.688297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9587.691324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9587.691488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9587.691501] Lustre: Skipped 4 previous similar messages [ 9587.696556] Lustre: Skipped 5 previous similar messages [ 9590.976345] Lustre: server umount lustre-MDT0000 complete [ 9592.662184] LustreError: 284621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878613 with bad export cookie 16055559367454858037 [ 9592.669271] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9592.670704] LustreError: 284621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9592.815351] Lustre: server umount lustre-MDT0001 complete [ 9604.759115] Lustre: server umount lustre-OST0000 complete [ 9616.760476] Lustre: server umount lustre-OST0001 complete [ 9620.415834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9620.568353] LustreError: 289376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9620.574600] LustreError: 289376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9622.210342] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9626.407576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9628.189932] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9631.118894] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9633.189412] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9635.392405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9637.262794] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9639.540604] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9655.189712] LustreError: 291206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9655.196209] LustreError: 291206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 9657.245085] loop1: detected capacity change from 0 to 204800000 [ 9657.314764] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9657.319334] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9657.323211] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9659.297868] loop1: detected capacity change from 0 to 204800000 [ 9661.174822] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9661.206175] LustreError: 289364:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs17018: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9661.210522] LustreError: 289364:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs17018 [ 9661.212460] LustreError: 289364:0:(mgs_handler.c:517:mgs_target_reg()) fs17018-MDT0001: Error checking target index: rc = -13 [ 9661.215460] LustreError: fs17018-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9661.219331] LustreError: 293109:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 9661.222286] LustreError: 293109:0:(tgt_mount.c:2132:server_put_super()) no obd fs17018-MDT0001 [ 9661.224857] LustreError: 293109:0:(tgt_mount.c:116:server_deregister_mount()) fs17018-MDT0001 not registered [ 9661.236627] Lustre: server umount fs17018-MDT0001 complete [ 9661.238791] LustreError: 293109:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9664.346767] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.388332] Lustre: Setting parameter fs17018-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs17018-MDT0001 [ 9664.392842] Lustre: Skipped 2 previous similar messages [ 9664.416211] Lustre: srv-fs17018-MDT0001: No data found on store. Initialize space. [ 9664.419040] Lustre: Skipped 4 previous similar messages [ 9664.467629] Lustre: fs17018-MDT0001: new disk, initializing [ 9664.469093] Lustre: Skipped 3 previous similar messages [ 9664.506067] Lustre: srv-fs17018-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9664.510057] Lustre: 293574:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17018-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9664.515485] LustreError: 293574:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17018-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9666.002991] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9669.617795] Lustre: server umount fs17018-MDT0001 complete [ 9674.958198] loop1: detected capacity change from 0 to 204800000 [ 9677.432932] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9677.612680] Lustre: 295139:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17018-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9677.618090] LustreError: 295139:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17018-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9679.152248] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9680.931189] Lustre: server umount fs17018-MDT0001 complete [ 9687.016322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.026229] Lustre: Skipped 3 previous similar messages [ 9691.419701] Lustre: server umount lustre-MDT0000 complete [ 9694.904885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9694.960815] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9696.721115] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9699.586206] loop1: detected capacity change from 0 to 204800000 [ 9700.339130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9701.692920] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9701.822544] Lustre: 297583:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17018-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9701.826119] LustreError: 297583:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17018-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9703.202480] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9705.534414] Lustre: server umount fs17018-MDT0001 complete [ 9715.686751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9720.805242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9720.805309] LustreError: 291206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9720.808923] Lustre: Skipped 11 previous similar messages [ 9720.819034] LustreError: 291206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 9721.273569] Lustre: server umount lustre-OST0000 complete [ 9737.183193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9737.267428] Lustre: server umount lustre-MDT0000 complete [ 9738.691967] LustreError: 290119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878759 with bad export cookie 16055559367454860718 [ 9738.700819] LustreError: 290119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9738.701011] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9738.867566] Lustre: server umount lustre-MDT0001 complete [ 9743.282805] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9744.500455] Key type lgssc unregistered [ 9744.660454] LNet: 299924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9744.663822] LNetError: 299924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9744.675455] LNet: Removed LNI 192.168.203.147@tcp [ 9745.111240] Key type .llcrypt unregistered [ 9745.113743] Key type ._llcrypt unregistered [ 9753.682890] Key type ._llcrypt registered [ 9753.684036] Key type .llcrypt registered [ 9753.727354] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9754.112695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9754.201648] alg: No test for adler32 (adler32-zlib) [ 9755.090258] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 9755.194887] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9756.783150] Key type lgssc registered [ 9757.325536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9760.922424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9760.933727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9762.139761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9764.135781] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9768.498750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9768.673695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9770.384635] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9773.754593] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9776.402589] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9779.574970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9779.772683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9781.889582] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9785.327086] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9787.949167] LustreError: 302994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9787.963223] LustreError: 302994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9791.884301] LustreError: 302994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.996860] LustreError: 302995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.018279] LustreError: 302996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.033201] LustreError: 302996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9802.136543] LustreError: 302996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9802.141255] LustreError: 302996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9803.449146] loop1: detected capacity change from 0 to 204800000 [ 9803.550451] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9803.556174] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9803.560781] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9805.790389] loop1: detected capacity change from 0 to 204800000 [ 9808.110096] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9808.167243] Lustre: fs23010-OST0000: new disk, initializing [ 9808.170602] Lustre: srv-fs23010-OST0000: No data found on store. Initialize space. [ 9808.175580] Lustre: 304918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9808.204867] Lustre: fs23010-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9810.418003] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9812.374681] LustreError: 302995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9812.380654] LustreError: 302995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9814.424749] Lustre: server umount fs23010-OST0000 complete [ 9819.922080] loop1: detected capacity change from 0 to 204800000 [ 9822.076677] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9822.141806] Lustre: fs23010-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9824.493594] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9828.619223] loop2: detected capacity change from 0 to 204800000 [ 9828.731613] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9828.736125] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9828.740140] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9829.864489] LustreError: 302996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9829.868524] LustreError: 302996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 9830.983374] loop2: detected capacity change from 0 to 204800000 [ 9833.439672] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9833.481487] LustreError: fs23010-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9833.485513] LustreError: fs23010-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9833.489834] LustreError: 307957:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 9833.492898] LustreError: 307957:0:(tgt_mount.c:2132:server_put_super()) no obd fs23010-OST0001 [ 9833.495584] LustreError: 307957:0:(tgt_mount.c:116:server_deregister_mount()) fs23010-OST0001 not registered [ 9833.509454] Lustre: server umount fs23010-OST0001 complete [ 9833.511174] LustreError: 307957:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9837.744287] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9837.832808] Lustre: fs23010-OST0001: new disk, initializing [ 9837.836389] Lustre: srv-fs23010-OST0001: No data found on store. Initialize space. [ 9837.842685] Lustre: 308509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9837.881172] Lustre: fs23010-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9840.522322] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9844.361050] Lustre: server umount fs23010-OST0000 complete [ 9848.189680] Lustre: server umount fs23010-OST0001 complete [ 9852.549870] loop1: detected capacity change from 0 to 204800000 [ 9854.579225] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9854.652889] Lustre: fs23010-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9857.271309] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9860.526169] loop2: detected capacity change from 0 to 204800000 [ 9862.742622] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9863.575612] LustreError: 302995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9863.588262] LustreError: 302995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 9865.391357] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9869.308100] Lustre: server umount fs23010-OST0000 complete [ 9870.822249] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9873.589284] Lustre: server umount fs23010-OST0001 complete [ 9880.032810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9880.038382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9880.048243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9881.058912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9881.062507] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9881.063197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9881.074478] LustreError: Skipped 1 previous similar message [ 9884.576652] Lustre: server umount lustre-OST0000 complete [ 9886.177650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.178972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9886.188318] Lustre: Skipped 1 previous similar message [ 9886.192211] Lustre: Skipped 1 previous similar message [ 9891.296934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9891.301499] Lustre: Skipped 1 previous similar message [ 9892.221148] Lustre: server umount lustre-MDT0000 complete [ 9893.432673] LustreError: 301147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878914 with bad export cookie 10057756677591394556 [ 9893.436523] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9893.437348] LustreError: 301147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9893.576377] Lustre: server umount lustre-MDT0001 complete [ 9897.142673] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9898.407857] Key type lgssc unregistered [ 9898.570599] LNet: 314419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9898.575520] LNetError: 314419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9898.584495] LNet: Removed LNI 192.168.203.147@tcp [ 9898.981083] Key type .llcrypt unregistered [ 9898.982750] Key type ._llcrypt unregistered [ 9910.314442] Key type ._llcrypt registered [ 9910.315604] Key type .llcrypt registered [ 9910.352237] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 9917.347501] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9917.668893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9917.677237] alg: No test for adler32 (adler32-zlib) [ 9918.525614] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 9918.632878] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9920.215102] Key type lgssc registered [ 9920.538034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9934.845586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9934.865589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9935.979719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9935.993239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9936.039263] Lustre: lustre-MDT0000: new disk, initializing [ 9936.065657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9936.073696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9937.636933] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9942.569812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9942.641575] Lustre: 318248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9942.681118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9942.685665] Lustre: Skipped 1 previous similar message [ 9942.731391] Lustre: lustre-MDT0001: new disk, initializing [ 9942.768592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9942.784762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9942.790953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9944.282724] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9948.529642] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9951.037035] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9954.003880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9954.149938] Lustre: lustre-OST0000: new disk, initializing [ 9954.153046] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9954.155782] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 9954.160350] Lustre: 320217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9954.197616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9956.331629] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9959.410788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9959.416339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9959.434647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9960.602506] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9961.448189] Lustre: Mounted lustre-client - version 2.17.57_117_g424a755 [ 9961.821940] Lustre: Unmounted lustre-client [ 9966.563971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9966.567521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9966.572708] Lustre: Skipped 1 previous similar message [ 9966.580064] Lustre: Skipped 1 previous similar message [ 9971.683733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9971.689287] Lustre: Skipped 1 previous similar message [ 9971.767934] Lustre: server umount lustre-OST0000 complete [ 9973.728596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9973.733819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9973.743209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9976.803426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.821147] Lustre: server umount lustre-MDT0000 complete [ 9981.363103] LustreError: 319042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879002 with bad export cookie 13748310427848907436 [ 9981.366535] LustreError: 319042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9981.366798] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9981.480181] Lustre: server umount lustre-MDT0001 complete [ 9985.320368] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9986.407893] Key type lgssc unregistered [10023.391485] LNet: 322375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10023.397511] LNetError: 322375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10094.053403] LNet: Removed LNI 192.168.203.147@tcp [10094.454522] Key type .llcrypt unregistered [10094.455908] Key type ._llcrypt unregistered [10101.595672] Key type ._llcrypt registered [10101.596674] Key type .llcrypt registered [10101.633876] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [10109.535280] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10109.966453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10109.990085] alg: No test for adler32 (adler32-zlib) [10110.912084] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [10111.033350] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [10112.647289] Key type lgssc registered [10113.286720] Lustre: Echo OBD driver; http://www.lustre.org/ [10125.988599] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:52:25 (1788879145) [10128.922324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10128.932155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.039227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10130.074443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10130.130074] Lustre: lustre-MDT0000: new disk, initializing [10130.153508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10130.160669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10131.886800] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10137.459411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10137.534193] Lustre: 325567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10137.582985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10137.587279] Lustre: Skipped 1 previous similar message [10137.639220] Lustre: lustre-MDT0001: new disk, initializing [10137.698242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10137.723535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10137.731739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10139.607319] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10144.507507] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10147.008193] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10149.788332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10149.880756] Lustre: lustre-OST0000: new disk, initializing [10149.882263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10149.884371] Lustre: 327535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10149.909644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10151.670650] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10155.181549] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10156.539201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10156.542617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10156.575768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10166.754594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10166.756269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10166.760374] LustreError: Skipped 1 previous similar message [10166.766574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10166.772557] Lustre: Skipped 1 previous similar message [10171.875487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10171.880022] Lustre: Skipped 1 previous similar message [10172.593088] Lustre: server umount lustre-OST0000 complete [10176.994404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10176.996136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10177.004924] Lustre: Skipped 2 previous similar messages [10180.599171] Lustre: server umount lustre-MDT0000 complete [10182.522635] LustreError: 325560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879203 with bad export cookie 7630263225199744574 [10182.530632] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10182.530628] LustreError: 325560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10182.692400] Lustre: server umount lustre-MDT0001 complete [10187.491704] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10188.838260] Key type lgssc unregistered [10188.972432] LNet: 329444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10188.976261] LNetError: 329444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10188.983327] LNet: Removed LNI 192.168.203.147@tcp [10189.335218] Key type .llcrypt unregistered [10189.336395] Key type ._llcrypt unregistered [10198.996906] Key type ._llcrypt registered [10198.998352] Key type .llcrypt registered [10199.042143] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10199.466449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10199.475946] alg: No test for adler32 (adler32-zlib) [10200.328499] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [10200.414698] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [10201.991109] Key type lgssc registered [10202.546898] Lustre: Echo OBD driver; http://www.lustre.org/ [10205.801381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10205.810696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10206.977046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10208.717296] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10212.579703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10212.734199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10214.521149] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10218.267512] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10220.766118] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10223.819212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10223.951191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10226.386693] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10229.318235] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10243.490670] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10244.124521] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10246.044721] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping) [10247.650309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10247.654914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10247.661719] Lustre: Skipped 2 previous similar messages [10247.665996] Lustre: Skipped 1 previous similar message [10251.158679] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping) [10252.208840] Lustre: server umount lustre-MDT0000 complete [10252.769198] LustreError: 330708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10252.775092] LustreError: 330708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10253.575341] LustreError: 330670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879274 with bad export cookie 5694665427898544859 [10253.579964] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10253.580593] LustreError: 330670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10256.278819] LustreError: 330684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.291780] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.47@tcp (stopping) [10257.889227] LustreError: 330686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10257.889692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10259.795278] Lustre: server umount lustre-MDT0001 complete [10267.697752] Lustre: server umount lustre-OST0000 complete [10271.652690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10273.311524] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10274.594169] Lustre: 334888:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.147@tcp [10275.623968] Lustre: server umount MGS complete [10279.465229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10279.654044] LustreError: 335458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10279.665218] LustreError: 335458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10279.685665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10281.210450] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10284.789525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.962677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10286.651341] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10290.420043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10293.086812] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10296.430305] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10297.096640] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10299.761757] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10307.880874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [10310.116095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10310.118439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10310.130996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10310.135665] Lustre: Skipped 3 previous similar messages [10311.535684] Lustre: server umount lustre-OST0000 complete [10315.234885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10315.242660] Lustre: Skipped 2 previous similar messages [10315.316937] Lustre: server umount lustre-MDT0000 complete [10317.247097] LustreError: 335438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879338 with bad export cookie 5694665427898545692 [10317.251328] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10317.255856] LustreError: 335438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10317.378976] Lustre: server umount lustre-MDT0001 complete [10321.908672] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10323.360491] Key type lgssc unregistered [10323.510469] LNet: 338890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10323.513876] LNetError: 338890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10323.522671] LNet: Removed LNI 192.168.203.147@tcp [10323.900859] Key type .llcrypt unregistered [10323.902145] Key type ._llcrypt unregistered [10331.327848] Key type ._llcrypt registered [10331.329093] Key type .llcrypt registered [10331.368688] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [10338.153973] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10338.451083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10338.460190] alg: No test for adler32 (adler32-zlib) [10339.303688] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [10339.387915] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [10340.959149] Key type lgssc registered [10341.479951] Lustre: Echo OBD driver; http://www.lustre.org/ [10357.410462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10357.418269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.516892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10358.531712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10358.583580] Lustre: lustre-MDT0000: new disk, initializing [10358.607288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10358.614241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10360.250098] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10365.701969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10365.762876] Lustre: 342018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10365.800695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10365.804980] Lustre: Skipped 1 previous similar message [10365.849668] Lustre: lustre-MDT0001: new disk, initializing [10365.883872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10365.897218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10365.903097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10367.674723] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10372.122906] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10374.014753] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10377.313357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10377.467393] Lustre: lustre-OST0000: new disk, initializing [10377.469853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10377.473302] Lustre: 343989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10377.509630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10380.066603] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10384.690591] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10386.279391] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10386.432437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10386.436924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10386.470254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10387.405060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10389.089565] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10389.157014] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10391.521900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10391.525710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.527424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10391.531346] Lustre: Skipped 1 previous similar message [10392.178086] Lustre: server umount lustre-OST0000 complete [10396.642695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10396.643237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10396.648715] LustreError: Skipped 1 previous similar message [10396.658401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10396.664144] Lustre: Skipped 2 previous similar messages [10399.916892] Lustre: server umount lustre-MDT0000 complete [10401.197356] LustreError: 342011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879422 with bad export cookie 18205773871277735245 [10401.200859] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10401.201051] LustreError: 342011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10401.334646] Lustre: server umount lustre-MDT0001 complete [10404.849305] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 10:57:04 (1788879424) [10412.407635] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [10415.912272] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10431.749183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10431.835792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10431.846584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10431.890636] Lustre: lustre-MDT0000: new disk, initializing [10431.914611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10431.922096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10433.559540] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10438.318630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10438.360490] Lustre: 348745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10438.387503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10438.389431] Lustre: Skipped 1 previous similar message [10438.425925] Lustre: lustre-MDT0001: new disk, initializing [10438.456335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10438.467082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10438.470575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10439.994654] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10444.307537] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10446.800172] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10450.112048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10450.240943] Lustre: lustre-OST0000: new disk, initializing [10450.242887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10450.245410] Lustre: 350715:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10450.289653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10452.760453] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10457.437259] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10459.651515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10459.658134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10459.694359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10474.977593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10474.981404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10474.982032] LustreError: Skipped 1 previous similar message [10474.987738] Lustre: Skipped 1 previous similar message [10474.988803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10474.994779] Lustre: Skipped 1 previous similar message [10480.096893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10480.098802] Lustre: Skipped 1 previous similar message [10480.144324] Lustre: server umount lustre-OST0000 complete [10484.703923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10484.706419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10484.709847] Lustre: Skipped 1 previous similar message [10484.711436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10487.874663] Lustre: server umount lustre-MDT0000 complete [10489.747837] LustreError: 348781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879510 with bad export cookie 18205773871277736113 [10489.754753] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10489.757468] LustreError: 348781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10489.896324] Lustre: server umount lustre-MDT0001 complete [10494.427221] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10495.811924] Key type lgssc unregistered [10495.986670] LNet: 352861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10495.992905] LNetError: 352861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10496.007613] LNet: Removed LNI 192.168.203.147@tcp [10496.441933] Key type .llcrypt unregistered [10496.443467] Key type ._llcrypt unregistered [10537.688872] Key type ._llcrypt registered [10537.690134] Key type .llcrypt registered [10537.728276] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10537.853239] Key type .llcrypt unregistered [10537.854778] Key type ._llcrypt unregistered [10574.285446] Key type ._llcrypt registered [10574.286912] Key type .llcrypt registered [10574.338456] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10574.511481] Key type .llcrypt unregistered [10574.513466] Key type ._llcrypt unregistered [10588.449891] Key type ._llcrypt registered [10588.451023] Key type .llcrypt registered [10588.505374] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10588.668333] Key type .llcrypt unregistered [10588.670748] Key type ._llcrypt unregistered [10602.594910] Key type ._llcrypt registered [10602.595945] Key type .llcrypt registered [10602.631975] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10602.748272] Key type .llcrypt unregistered [10602.749311] Key type ._llcrypt unregistered [10639.206396] Key type ._llcrypt registered [10639.207684] Key type .llcrypt registered [10639.258710] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10639.400297] Key type .llcrypt unregistered [10639.401463] Key type ._llcrypt unregistered [10653.098176] Key type ._llcrypt registered [10653.099225] Key type .llcrypt registered [10653.132308] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10653.240342] Key type .llcrypt unregistered [10653.241826] Key type ._llcrypt unregistered [10667.240348] Key type ._llcrypt registered [10667.241334] Key type .llcrypt registered [10667.279550] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10667.375182] Key type .llcrypt unregistered [10667.376625] Key type ._llcrypt unregistered [10680.914383] Key type ._llcrypt registered [10680.916010] Key type .llcrypt registered [10680.965096] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10681.082636] Key type .llcrypt unregistered [10681.084404] Key type ._llcrypt unregistered [10694.904915] Key type ._llcrypt registered [10694.907326] Key type .llcrypt registered [10694.967913] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10695.147842] Key type .llcrypt unregistered [10695.150245] Key type ._llcrypt unregistered [10707.059868] Key type ._llcrypt registered [10707.061727] Key type .llcrypt registered [10707.108274] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10707.238316] Key type .llcrypt unregistered [10707.240461] Key type ._llcrypt unregistered [10736.481984] Key type ._llcrypt registered [10736.483167] Key type .llcrypt registered [10736.522903] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10736.626234] Key type .llcrypt unregistered [10736.627623] Key type ._llcrypt unregistered [10751.998658] Key type ._llcrypt registered [10751.999793] Key type .llcrypt registered [10752.040613] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10752.157479] Key type .llcrypt unregistered [10752.161156] Key type ._llcrypt unregistered [10765.774964] Key type ._llcrypt registered [10765.775949] Key type .llcrypt registered [10765.807465] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10765.894162] Key type .llcrypt unregistered [10765.895521] Key type ._llcrypt unregistered [10795.976756] Key type ._llcrypt registered [10795.978288] Key type .llcrypt registered [10796.038385] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [10804.449428] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10804.820203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10804.826964] alg: No test for adler32 (adler32-zlib) [10805.748972] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [10805.885364] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [10807.495201] Key type lgssc registered [10808.108934] Lustre: Echo OBD driver; http://www.lustre.org/ [10822.240162] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10822.244514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10823.323786] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10823.332600] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10823.367614] Lustre: tst250fs-MDT0000: new disk, initializing [10823.386127] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10823.391913] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10824.698427] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10830.682389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10830.726272] Lustre: 372747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10830.752724] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10830.754696] Lustre: Skipped 1 previous similar message [10830.792754] Lustre: tst250fs-MDT0001: new disk, initializing [10830.829354] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10830.844052] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10830.847569] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10832.161187] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10834.358728] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10837.701205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10837.804689] Lustre: tst250fs-OST0000: new disk, initializing [10837.806165] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10837.808627] Lustre: 374381:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10837.838407] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10839.919267] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10843.657408] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10843.664699] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10843.720239] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10844.696161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10844.758695] Lustre: tst250fs-OST0001: new disk, initializing [10844.760398] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10844.762907] Lustre: 375250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10844.794917] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10846.834620] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10849.524939] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10853.320592] Lustre: 372752:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.147@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [10861.536086] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10861.538786] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10861.543479] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10865.634446] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10865.634809] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10865.637855] Lustre: Skipped 1 previous similar message [10865.640509] Lustre: Skipped 3 previous similar messages [10867.579502] Lustre: server umount tst250fs-MDT0000 complete [10869.986476] LustreError: 372742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879890 with bad export cookie 6184685031995794365 [10869.992201] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10870.121085] Lustre: server umount tst250fs-MDT0001 complete [10881.690863] Lustre: server umount tst250fs-OST0000 complete [10893.936114] Lustre: server umount tst250fs-OST0001 complete [10902.165705] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [10905.210261] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10916.140945] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:05:36 (1788879936) [10916.647325] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10922.643054] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [10927.573307] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10941.043131] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10579 sec ========= 11:06:01 (1788879961) [10941.846239] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:06:01 (1788879961) === [10943.549475] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:06:03 (1788879963) === [10957.998270] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10959.047950] Key type lgssc unregistered [10959.205689] LNet: 384356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10959.209680] LNetError: 384356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10959.219508] LNet: Removed LNI 192.168.203.147@tcp [10959.748260] Key type .llcrypt unregistered [10959.751098] Key type ._llcrypt unregistered