[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480950631 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003151] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008340] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022810] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027405] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031029] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032117] *** VALIDATE tmpfs *** [ 0.033503] *** VALIDATE proc *** [ 0.035076] *** VALIDATE cgroup *** [ 0.036012] *** VALIDATE cgroup2 *** [ 0.038006] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041032] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045074] debug: unmapping init [mem 0xffffffff8da59000-0xffffffff8da60fff] [ 0.047174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050009] ... bit width: 48 [ 0.050883] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054017] ... event mask: 000000070000000f [ 0.055362] rcu: Hierarchical SRCU implementation. [ 0.057532] smp: Bringing up secondary CPUs ... [ 0.058665] x86: Booting SMP configuration: [ 0.059030] .... node #0, CPUs: #1 #2 #3 [ 0.062572] smp: Brought up 1 node, 4 CPUs [ 0.064012] smpboot: Max logical packages: 1 [ 0.065022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136344] node 0 deferred pages initialised in 70ms [ 0.140097] devtmpfs: initialized [ 0.141349] x86/mm: Memory block size: 128MB [ 0.145433] gcov: version magic: 0x41383552 [ 0.148359] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150284] pinctrl core: initialized pinctrl subsystem [ 0.151191] [ 0.151759] ************************************************************* [ 0.152019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153016] ** ** [ 0.154015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155018] ** ** [ 0.156013] ** This means that this kernel is built to expose internal ** [ 0.157019] ** IOMMU data structures, which may compromise security on ** [ 0.158024] ** your system. ** [ 0.159015] ** ** [ 0.160015] ** If you see this message and you are not debugging the ** [ 0.161013] ** kernel, report this immediately to your vendor! ** [ 0.162016] ** ** [ 0.163015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164015] ************************************************************* [ 0.165725] NET: Registered protocol family 16 [ 0.166483] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169455] cpuidle: using governor menu [ 0.172228] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174520] PCI: Using configuration type 1 for base access [ 0.176133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187090] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.190039] cryptd: max_cpu_qlen set to 1000 [ 0.192276] ACPI: Added _OSI(Module Device) [ 0.193018] ACPI: Added _OSI(Processor Device) [ 0.194012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198944] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200581] ACPI: Interpreter enabled [ 0.201071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202015] ACPI: Using IOAPIC for interrupt routing [ 0.203101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.204426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.212700] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.217080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.221076] acpiphp: Slot [2] registered [ 0.222087] acpiphp: Slot [5] registered [ 0.223102] acpiphp: Slot [6] registered [ 0.224103] acpiphp: Slot [7] registered [ 0.225069] acpiphp: Slot [8] registered [ 0.226038] acpiphp: Slot [9] registered [ 0.227008] acpiphp: Slot [10] registered [ 0.227915] acpiphp: Slot [3] registered [ 0.229057] acpiphp: Slot [4] registered [ 0.230080] acpiphp: Slot [11] registered [ 0.231020] acpiphp: Slot [12] registered [ 0.231921] acpiphp: Slot [13] registered [ 0.233059] acpiphp: Slot [14] registered [ 0.234080] acpiphp: Slot [15] registered [ 0.235048] acpiphp: Slot [16] registered [ 0.235954] acpiphp: Slot [17] registered [ 0.237075] acpiphp: Slot [18] registered [ 0.238065] acpiphp: Slot [19] registered [ 0.239045] acpiphp: Slot [20] registered [ 0.239987] acpiphp: Slot [21] registered [ 0.240090] acpiphp: Slot [22] registered [ 0.242093] acpiphp: Slot [23] registered [ 0.243110] acpiphp: Slot [24] registered [ 0.245136] acpiphp: Slot [25] registered [ 0.246073] acpiphp: Slot [26] registered [ 0.246995] acpiphp: Slot [27] registered [ 0.248059] acpiphp: Slot [28] registered [ 0.249061] acpiphp: Slot [29] registered [ 0.250059] acpiphp: Slot [30] registered [ 0.251066] acpiphp: Slot [31] registered [ 0.252042] PCI host bridge to bus 0000:00 [ 0.252927] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.254020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.256013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.257015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.259069] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.260030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.262249] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.263740] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.265905] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.272017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.276054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.278018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.280012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.281010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.283452] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.284519] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.287048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.291012] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.296014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.308014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.314012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.318643] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.330018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.339018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.355015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.366122] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.371015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.376019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.390014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.399011] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.404026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.409016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.431017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.437025] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.445031] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.453018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.468013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.478583] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.484015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.496017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.507017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.514878] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.525017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.531017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.545018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.555664] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.558284] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.560278] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.561206] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.563202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.566031] iommu: Default domain type: Passthrough [ 0.568384] SCSI subsystem initialized [ 0.570133] ACPI: bus type USB registered [ 0.571139] usbcore: registered new interface driver usbfs [ 0.574104] usbcore: registered new interface driver hub [ 0.576072] usbcore: registered new device driver usb [ 0.578214] pps_core: LinuxPPS API ver. 1 registered [ 0.581014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.584061] PTP clock support registered [ 0.587115] EDAC MC: Ver: 3.0.0 [ 0.588331] PCI: Using ACPI for IRQ routing [ 0.589627] NetLabel: Initializing [ 0.590007] NetLabel: domain hash size = 128 [ 0.590801] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.592083] NetLabel: unlabeled traffic allowed by default [ 0.595054] vgaarb: loaded [ 0.596258] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.598014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.603280] clocksource: Switched to clocksource kvm-clock [ 0.707920] VFS: Disk quotas dquot_6.6.0 [ 0.712246] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.715987] *** VALIDATE ramfs *** [ 0.718062] *** VALIDATE hugetlbfs *** [ 0.720484] pnp: PnP ACPI init [ 0.723516] pnp: PnP ACPI: found 6 devices [ 0.765632] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.770271] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.772849] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.774991] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.777898] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.780181] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.783297] NET: Registered protocol family 2 [ 0.786088] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.791013] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.794625] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.800275] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.803829] TCP: Hash tables configured (established 65536 bind 65536) [ 0.806747] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.810794] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.814421] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.817358] NET: Registered protocol family 1 [ 0.820860] RPC: Registered named UNIX socket transport module. [ 0.823152] RPC: Registered udp transport module. [ 0.825140] RPC: Registered tcp transport module. [ 0.827072] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.829622] NET: Registered protocol family 44 [ 0.831506] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.834027] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.836326] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.839153] PCI: CLS 0 bytes, default 64 [ 0.840963] Unpacking initramfs... [ 2.219864] debug: unmapping init [mem 0xffff8cbd3cc54000-0xffff8cbd3ffbffff] [ 2.223671] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.225812] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.228718] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.712576] Initialise system trusted keyrings [ 2.714211] Key type blacklist registered [ 2.716077] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.724234] zbud: loaded [ 2.726892] *** VALIDATE nfs *** [ 2.727823] *** VALIDATE nfs4 *** [ 2.729551] pstore: using deflate compression [ 2.733322] Platform Keyring initialized [ 2.837265] NET: Registered protocol family 38 [ 2.839149] Key type asymmetric registered [ 2.840853] Asymmetric key parser 'x509' registered [ 2.842510] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.844973] io scheduler mq-deadline registered [ 2.847513] io scheduler kyber registered [ 2.849938] io scheduler bfq registered [ 2.852600] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.857174] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.860765] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.865118] ACPI: Power Button [PWRF] [ 2.871350] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.878148] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.892418] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.898574] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.919593] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.947995] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.976407] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.982273] Non-volatile memory driver v1.3 [ 2.984608] Linux agpgart interface v0.103 [ 3.015929] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.019220] vda: detected capacity change from 0 to 74825728 [ 3.033970] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.035650] vdb: detected capacity change from 0 to 1073741824 [ 3.053944] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.057540] vdc: detected capacity change from 0 to 2621440000 [ 3.072852] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.075650] vdd: detected capacity change from 0 to 2621440000 [ 3.099081] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.103548] vde: detected capacity change from 0 to 4294967296 [ 3.128112] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.131471] vdf: detected capacity change from 0 to 4294967296 [ 3.137326] libphy: Fixed MDIO Bus: probed [ 3.144043] usbcore: registered new interface driver usbserial_generic [ 3.146933] usbserial: USB Serial support registered for generic [ 3.149768] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.154088] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.156054] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.158693] mousedev: PS/2 mouse device common for all mice [ 3.161765] rtc_cmos 00:05: RTC can wake from S4 [ 3.165238] rtc_cmos 00:05: registered as rtc0 [ 3.166353] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.167314] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.173743] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.174678] intel_pstate: CPU model not supported [ 3.178565] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.183406] hid: raw HID events driver (C) Jiri Kosina [ 3.185370] usbcore: registered new interface driver usbhid [ 3.187337] usbhid: USB HID core driver [ 3.189074] drop_monitor: Initializing network drop monitor service [ 3.191554] Initializing XFRM netlink socket [ 3.193647] NET: Registered protocol family 10 [ 3.197582] Segment Routing with IPv6 [ 3.198942] NET: Registered protocol family 17 [ 3.200806] mpls_gso: MPLS GSO support [ 3.205908] RAS: Correctable Errors collector initialized. [ 3.207748] AVX version of gcm_enc/dec engaged. [ 3.208919] AES CTR mode by8 optimization enabled [ 3.292911] sched_clock: Marking stable (3292868661, 0)->(4214819988, -921951327) [ 3.296547] registered taskstats version 1 [ 3.298771] Loading compiled-in X.509 certificates [ 3.300536] zswap: loaded using pool lzo/zbud [ 3.327181] Key type big_key registered [ 3.341242] Key type encrypted registered [ 3.343511] ima: No TPM chip found, activating TPM-bypass! [ 3.346536] ima: Allocated hash algorithm: sha1 [ 3.349037] ima: No architecture policies found [ 3.351502] evm: Initialising EVM extended attributes: [ 3.354348] evm: security.selinux [ 3.356263] evm: security.ima [ 3.357871] evm: security.capability [ 3.359768] evm: HMAC attrs: 0x1 [ 3.362766] rtc_cmos 00:05: setting system clock to 2026-08-24 23:13:46 UTC (1787613226) [ 3.369579] debug: unmapping init [mem 0xffffffff8ea03000-0xffffffff8ebfffff] [ 3.373884] debug: unmapping init [mem 0xffffffff8d782000-0xffffffff8da58fff] [ 3.384071] Write protecting the kernel read-only data: 28672k [ 3.387702] debug: unmapping init [mem 0xffffffff8be03000-0xffffffff8bffffff] [ 3.391966] debug: unmapping init [mem 0xffffffff8c714000-0xffffffff8c7fffff] [ 3.427844] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.438407] systemd[1]: Detected virtualization kvm. [ 3.440593] systemd[1]: Detected architecture x86-64. [ 3.443154] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.466798] systemd[1]: No hostname configured. [ 3.468990] systemd[1]: Set hostname to . [ 3.471696] random: systemd: uninitialized urandom read (16 bytes read) [ 3.475057] systemd[1]: Initializing machine ID from random generator. [ 3.508726] random: ln: uninitialized urandom read (6 bytes read) [ 3.615387] random: systemd: uninitialized urandom read (16 bytes read) [ 3.617266] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.622801] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.626751] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.280248] device-mapper: uevent: version 1.0.3 [ 4.282826] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.031003] random: fast init done [ 5.034382] virtio_net virtio0 ens2: renamed from eth0 [ 5.114799] scsi host0: ata_piix [ 5.149742] scsi host1: ata_piix [ 5.151601] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.154208] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.866585] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.865502] random: crng init done [ 9.867070] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.341026] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.556200] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.808833] SELinux: Disabled at runtime. [ 11.868065] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.874214] systemd[1]: Detected virtualization kvm. [ 11.876053] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.414984] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.417601] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.422198] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.427400] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.430974] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.442852] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.448760] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ 12.505088] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.013806] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.318211] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.370808] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.502086] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.510879] EDAC sbridge: Ver: 1.1.2 [ 15.119164] Key type dns_resolver registered [ 15.418489] NFS: Registering the id_resolver key type [ 15.420171] Key type id_resolver registered [ 15.421446] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg237-server login: [ 42.061565] libcfs: loading out-of-tree module taints kernel. [ 42.084633] Key type ._llcrypt registered [ 42.086306] Key type .llcrypt registered [ 42.144277] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 59.145273] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 60.397145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.404879] alg: No test for adler32 (adler32-zlib) [ 61.593420] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 62.396155] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 64.192173] Key type lgssc registered [ 65.880657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.641818] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 101.639785] hrtimer: interrupt took 4425660 ns [ 138.102850] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 154.101743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.135513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.543816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.595298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.759883] Lustre: lustre-MDT0000: new disk, initializing [ 155.905918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.951272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.490518] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.621591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.801784] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.871715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.880465] Lustre: Skipped 1 previous similar message [ 178.015721] Lustre: lustre-MDT0001: new disk, initializing [ 178.135430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.181218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.208877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.203492] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.182290] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.318811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.640265] Lustre: lustre-OST0000: new disk, initializing [ 198.643523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.651461] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.714280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.456047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.464798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.535301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.134271] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.159904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.281452] Lustre: lustre-OST0001: new disk, initializing [ 220.287440] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.293686] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.387271] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.781403] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.387621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.400731] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.443881] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 239.161468] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.780412] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 258.804695] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing check_logdir /tmp/testlogs/ [ 264.155655] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing yml_node [ 268.023238] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 271.033159] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 274.411758] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 276.322458] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 19:18:17 EDT 2026 [ 298.511758] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 300.204942] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.481212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.486032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.495213] Lustre: Skipped 1 previous similar message [ 320.520343] Lustre: Skipped 3 previous similar messages [ 323.413522] Lustre: server umount lustre-MDT0000 complete [ 327.804359] LustreError: 7457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787613550 with bad export cookie 5408901782595310460 [ 327.814563] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 327.816211] LustreError: 7457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 330.720787] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.721298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.722850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.730145] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 330.751927] Lustre: Skipped 3 previous similar messages [ 332.774493] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.775041] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.791052] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 332.794841] Lustre: Skipped 2 previous similar messages [ 334.070261] Lustre: server umount lustre-MDT0001 complete [ 338.830893] Lustre: server umount lustre-OST0000 complete [ 343.551143] Lustre: server umount lustre-OST0001 complete [ 351.667338] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 363.618090] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 406.410736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.818744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 406.880559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 406.988567] Lustre: lustre-MDT0000: new disk, initializing [ 407.133455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.152739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 413.050892] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 427.606421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.891444] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 428.033514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 428.040939] Lustre: Skipped 1 previous similar message [ 428.230136] Lustre: lustre-MDT0001: new disk, initializing [ 428.407033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 428.461922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 428.487783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 433.343097] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 445.624364] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 453.686200] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 461.199443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 461.638250] Lustre: lustre-OST0000: new disk, initializing [ 461.644408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 461.663929] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 461.794351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.917836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 462.922927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 463.043376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 468.552958] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 479.003471] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 483.757568] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 483.986943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 489.448358] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 489.841462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 494.057630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.058213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.102493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.108778] Lustre: Skipped 1 previous similar message [ 498.732402] Lustre: server umount lustre-OST0000 complete [ 504.290902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.299668] Lustre: Skipped 1 previous similar message [ 504.312169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 508.412538] Lustre: server umount lustre-MDT0000 complete [ 509.411891] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.439389] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 512.154367] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787613735 with bad export cookie 5408901782595311993 [ 512.172851] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.183263] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 512.716484] Lustre: server umount lustre-MDT0001 complete [ 521.302258] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:22:22 (1787613742) [ 542.183959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.665763] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.766308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.211401] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 547.812225] LustreError: 19924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.761947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.036984] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 557.046420] Lustre: Skipped 1 previous similar message [ 562.138313] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 569.924603] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 576.004512] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 578.502580] Lustre: Failing over lustre-MDT0000 [ 578.807109] Lustre: server umount lustre-MDT0000 complete [ 582.625587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 582.639416] LustreError: Skipped 1 previous similar message [ 582.642559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.647175] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.655139] Lustre: Skipped 2 previous similar messages [ 582.686758] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 597.985356] Lustre: 3642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787613805/real 1787613805] req@ffff8cbdb6075f80 x1874448388098304/t0(0) o400->MGC192.168.202.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787613821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 598.019202] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.092372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.145439] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.152232] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 609.511645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.518302] Lustre: Skipped 1 previous similar message [ 609.570195] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 614.890461] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 614.897991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 614.943812] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 615.558180] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 627.019981] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 632.299310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.305281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.330470] Lustre: Skipped 1 previous similar message [ 632.356739] Lustre: Skipped 1 previous similar message [ 638.096059] Lustre: server umount lustre-MDT0000 complete [ 642.449682] LustreError: 21491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787613865 with bad export cookie 5408901782595313407 [ 642.476567] LustreError: 21491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 642.478384] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 642.821702] Lustre: server umount lustre-MDT0001 complete [ 651.454172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 652.233829] LustreError: 23263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.264589] LustreError: 23263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 657.340694] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 666.682208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.956368] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 681.654494] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 688.331442] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 690.957709] Lustre: Failing over lustre-MDT0000 [ 691.357595] Lustre: server umount lustre-MDT0000 complete [ 692.010067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.580858] Lustre: 3640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787613915/real 1787613915] req@ffff8cbc867fad80 x1874448388128640/t0(0) o400->MGC192.168.202.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787613931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 708.602381] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 711.713640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 718.823204] LustreError: 23260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.834375] LustreError: 23260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 719.841547] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cbdbf702300 x1874448388131840/t0(0) o250->MGC192.168.202.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 720.245291] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 725.477720] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 725.490333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 725.501351] Lustre: Skipped 1 previous similar message [ 725.514812] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 725.786905] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 736.731673] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 742.889531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.894148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.920212] Lustre: Skipped 2 previous similar messages [ 742.924635] Lustre: Skipped 2 previous similar messages [ 748.543326] Lustre: server umount lustre-MDT0000 complete [ 751.918696] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787613975 with bad export cookie 5408901782595314345 [ 751.929928] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 751.930356] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 752.158714] Lustre: server umount lustre-MDT0001 complete [ 761.899526] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 19:26:23 (1787613983) [ 781.252645] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 791.627695] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 839.117587] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 851.306961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.584177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 851.599566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 851.652869] Lustre: lustre-MDT0000: new disk, initializing [ 851.772364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 851.776176] Lustre: Skipped 3 previous similar messages [ 851.793453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 856.403439] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 867.911262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.044805] Lustre: 29645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 868.405252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 872.772824] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 883.614173] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 892.043412] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 900.704497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 901.126684] Lustre: 31614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 902.461968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 902.516974] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 902.617091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 902.646882] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 909.149629] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 920.422732] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 927.130092] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 953.837619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.844778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.862146] Lustre: Skipped 1 previous similar message [ 953.862695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.878519] Lustre: Skipped 3 previous similar messages [ 967.136138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 967.407472] Lustre: server umount lustre-OST0000 complete [ 986.080379] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 986.387369] Lustre: server umount lustre-MDT0000 complete [ 989.670191] LustreError: 31684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.700424] LustreError: 31684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 990.484033] LustreError: 30448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614213 with bad export cookie 5408901782595314751 [ 990.490574] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 990.497992] LustreError: 30448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 991.133963] Lustre: server umount lustre-MDT0001 complete [ 1003.899536] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1006.924777] Key type lgssc unregistered [ 1007.352875] LNet: 33600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.369970] LNetError: 33600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.384264] LNet: Removed LNI 192.168.202.137@tcp [ 1008.361269] Key type .llcrypt unregistered [ 1008.365072] Key type ._llcrypt unregistered [ 1033.812497] Key type ._llcrypt registered [ 1033.814563] Key type .llcrypt registered [ 1033.906412] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1035.202514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.230178] alg: No test for adler32 (adler32-zlib) [ 1036.464120] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 1036.806053] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1038.448388] Key type lgssc registered [ 1040.015142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1049.979987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1050.002853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.563793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.134371] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1060.898936] LustreError: 35018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1060.926717] LustreError: 35018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1065.984312] LustreError: 35019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1067.997307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.498837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.336485] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1083.161801] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1090.467063] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1097.510673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1097.839924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.460859] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1112.283379] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1113.210433] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1121.436424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1137.525235] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:32:38 (1787614358) [ 1139.931415] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1141.982247] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1150.437143] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:32:51 (1787614371) [ 1164.761803] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:33:05 (1787614385) [ 1166.131184] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1175.692224] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1175.702591] Lustre: Skipped 2 previous similar messages [ 1177.434313] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:33:19 (1787614399) [ 1180.072752] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1180.086844] Lustre: Skipped 2 previous similar messages [ 1189.581634] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1189.593454] Lustre: Skipped 4 previous similar messages [ 1207.374639] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:33:48 (1787614428) [ 1211.185836] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1216.727660] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1230.397307] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:34:11 (1787614451) [ 1232.594053] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1232.604258] Lustre: Skipped 6 previous similar messages [ 1248.837580] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:34:29 (1787614469) [ 1265.154397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.163234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1265.175053] Lustre: Skipped 1 previous similar message [ 1265.177079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1270.255233] LustreError: 36840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.270883] LustreError: 36840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1270.509681] Lustre: server umount lustre-OST0000 complete [ 1275.371836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.384183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1275.395218] Lustre: Skipped 1 previous similar message [ 1275.421038] Lustre: Skipped 2 previous similar messages [ 1280.482336] LustreError: 35019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1280.490714] Lustre: server umount lustre-MDT0000 complete [ 1284.454688] LustreError: 39467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614507 with bad export cookie 13858880003777617943 [ 1284.459138] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1284.472864] LustreError: 39467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1284.929966] Lustre: server umount lustre-MDT0001 complete [ 1298.564923] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1301.310166] Key type lgssc unregistered [ 1301.638181] LNet: 42904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.642943] LNetError: 42904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1302.698862] LNet: Removed LNI 192.168.202.137@tcp [ 1303.885609] Key type .llcrypt unregistered [ 1303.888420] Key type ._llcrypt unregistered [ 1323.917401] Key type ._llcrypt registered [ 1323.919803] Key type .llcrypt registered [ 1324.017430] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 1338.410345] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1339.990336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1340.053903] alg: No test for adler32 (adler32-zlib) [ 1341.273325] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 1341.745058] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1343.528156] Key type lgssc registered [ 1345.206780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1383.761307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1383.772542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.047724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1385.073520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1385.157199] Lustre: lustre-MDT0000: new disk, initializing [ 1385.238874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.272258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1390.465985] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1400.571883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.715369] Lustre: 46060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1400.753778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1400.759711] Lustre: Skipped 1 previous similar message [ 1400.823965] Lustre: lustre-MDT0001: new disk, initializing [ 1400.919944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1400.944111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1400.957578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1406.137356] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1418.708566] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1424.885511] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1432.801109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1433.031413] Lustre: lustre-OST0000: new disk, initializing [ 1433.044970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1433.054325] Lustre: 48028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1433.178829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.220183] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1442.940908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1442.953991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1443.113211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1448.790969] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1453.402563] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1453.623372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1458.075436] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1458.280071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1463.786188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1463.787181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.794682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1463.796779] LustreError: Skipped 1 previous similar message [ 1463.828384] Lustre: Skipped 1 previous similar message [ 1466.258414] Lustre: server umount lustre-OST0000 complete [ 1472.487714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1472.497463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.511589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1472.514071] Lustre: Skipped 1 previous similar message [ 1474.017794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1474.029591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.255735] Lustre: server umount lustre-MDT0000 complete [ 1480.008762] LustreError: 46053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614703 with bad export cookie 1515303339525260477 [ 1480.013843] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.020533] LustreError: 46053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1480.275677] Lustre: server umount lustre-MDT0001 complete [ 1488.985104] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:38:30 (1787614710) [ 1495.367372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1495.740488] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.764685] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1495.842923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.637912] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1501.154393] LustreError: 50492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.250433] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.375375] LustreError: 50492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.426247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.847162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1514.533490] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1521.877644] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1528.152546] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1535.404419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.470778] LustreError: 52313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.848129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.193534] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1550.392114] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1555.075658] LustreError: 53020:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1555.951288] Lustre: Modifying parameter general.timeout=1 in log params [ 1557.785339] Lustre: Modifying parameter general.timeout=3 in log params [ 1557.787589] Lustre: Skipped 1 previous similar message [ 1560.342729] Lustre: Modifying parameter general.timeout=6 in log params [ 1560.349159] Lustre: Skipped 2 previous similar messages [ 1564.372669] Lustre: Modifying parameter general.timeout=10 in log params [ 1564.378584] Lustre: Skipped 3 previous similar messages [ 1573.209713] Lustre: Modifying parameter general.timeout=19 in log params [ 1573.217327] Lustre: Skipped 8 previous similar messages [ 1589.767675] Lustre: Modifying parameter general.timeout=37 in log params [ 1589.778658] Lustre: Skipped 17 previous similar messages [ 1622.491938] Lustre: Modifying parameter general.timeout=72 in log params [ 1622.497992] Lustre: Skipped 34 previous similar messages [ 1682.952241] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:41:44 (1787614904) [ 1686.176715] Lustre: 59972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1687.313614] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1687.321376] Lustre: Skipped 60 previous similar messages [ 1688.468143] Lustre: 60071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1692.064548] Lustre: 60267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1692.084856] Lustre: 60267:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1696.807143] Lustre: 60484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1696.834614] Lustre: 60484:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1727.127691] Lustre: 61841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1727.144333] LustreError: 61841:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1731.710059] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:42:32 (1787614952) [ 1746.401340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1746.418810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.448916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.463814] Lustre: Skipped 1 previous similar message [ 1750.010684] Lustre: server umount lustre-MDT0000 complete [ 1753.569631] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.589469] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1753.991915] LustreError: 50473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614977 with bad export cookie 1515303339525261359 [ 1753.993433] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1754.001328] LustreError: 50473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1754.561752] Lustre: server umount lustre-MDT0001 complete [ 1769.737972] Lustre: server umount lustre-OST0000 complete [ 1802.912628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.307541] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1803.672324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.423263] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1817.473928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.555215] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1817.582589] Lustre: 64374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1822.476414] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1831.685687] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1838.330603] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1845.281299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1845.504521] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1845.845340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.848755] Lustre: Skipped 1 previous similar message [ 1852.011453] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1860.065290] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1861.922152] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1861.930712] Lustre: Skipped 20 previous similar messages [ 1862.029416] Lustre: 66942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1873.889970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1873.902774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.917814] Lustre: Skipped 2 previous similar messages [ 1873.931778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1873.940741] Lustre: Skipped 3 previous similar messages [ 1876.304960] Lustre: server umount lustre-OST0000 complete [ 1884.651574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.657262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1884.668403] Lustre: Skipped 1 previous similar message [ 1884.676718] LustreError: Skipped 1 previous similar message [ 1884.683287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.694548] Lustre: Skipped 3 previous similar messages [ 1886.802399] Lustre: server umount lustre-MDT0000 complete [ 1889.765870] LustreError: 65400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.788892] LustreError: 65400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1891.231302] LustreError: 64368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615114 with bad export cookie 1515303339525264467 [ 1891.248111] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1891.250742] LustreError: 64368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1891.858417] Lustre: server umount lustre-MDT0001 complete [ 1904.567849] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1907.867147] Key type lgssc unregistered [ 1908.194986] LNet: 68160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.200379] LNetError: 68160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.220498] LNet: Removed LNI 192.168.202.137@tcp [ 1909.278446] Key type .llcrypt unregistered [ 1909.282433] Key type ._llcrypt unregistered [ 1935.206305] Key type ._llcrypt registered [ 1935.207901] Key type .llcrypt registered [ 1935.325808] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1936.434559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1936.496798] alg: No test for adler32 (adler32-zlib) [ 1937.615597] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 1937.822605] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1939.504220] Key type lgssc registered [ 1940.656945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1949.347460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1949.384258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.917328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.663801] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1960.436879] LustreError: 69577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.459420] LustreError: 69577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1965.460730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.546327] LustreError: 69576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.205536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1971.457596] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1980.544781] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1987.408389] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1995.409190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1996.199267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.730935] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2012.837303] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2022.464671] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2032.609262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2032.622901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.647654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.196497] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.203808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.224451] Lustre: Skipped 1 previous similar message [ 2036.239430] Lustre: Skipped 2 previous similar messages [ 2038.118606] Lustre: server umount lustre-MDT0000 complete [ 2041.315227] LustreError: 69576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.352741] LustreError: 69576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2042.118280] LustreError: 69557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615265 with bad export cookie 10678798584462761370 [ 2042.130770] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2042.136810] LustreError: 69557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2042.406872] Lustre: server umount lustre-MDT0001 complete [ 2057.653260] Lustre: server umount lustre-OST0000 complete [ 2086.581292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.949360] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2087.020269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2087.459420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.799628] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2101.060780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.150791] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2101.197842] Lustre: 74451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2101.718259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2106.650660] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2115.126566] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2121.531834] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2130.380830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.706734] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2131.192992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.176923] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2154.185662] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2156.547785] Lustre: 77030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2159.508620] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2168.802172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2168.815623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.860794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2172.273616] Lustre: server umount lustre-OST0000 complete [ 2178.529477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2178.541190] LustreError: Skipped 1 previous similar message [ 2178.543028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.563187] Lustre: Skipped 1 previous similar message [ 2178.588274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.599789] Lustre: Skipped 2 previous similar messages [ 2182.667454] Lustre: server umount lustre-MDT0000 complete [ 2184.673942] LustreError: 74461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.689637] LustreError: 74461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2187.044181] LustreError: 76334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615410 with bad export cookie 10678798584462762203 [ 2187.051209] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2187.058622] LustreError: 76334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2187.574485] Lustre: server umount lustre-MDT0001 complete [ 2200.869261] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2204.571926] Key type lgssc unregistered [ 2204.902819] LNet: 78242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.908442] LNetError: 78242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.947339] LNet: Removed LNI 192.168.202.137@tcp [ 2206.167840] Key type .llcrypt unregistered [ 2206.174336] Key type ._llcrypt unregistered [ 2235.077869] Key type ._llcrypt registered [ 2235.079987] Key type .llcrypt registered [ 2235.260778] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2236.422811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2236.597153] alg: No test for adler32 (adler32-zlib) [ 2237.844082] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 2238.197410] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2239.928197] Key type lgssc registered [ 2241.517495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2252.274881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2252.291623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.036316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.355387] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2264.545888] LustreError: 79656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.576174] LustreError: 79656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2269.684025] LustreError: 79657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.880054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2270.754194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2275.951700] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2284.589149] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2291.885705] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2300.719046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2301.458337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.765888] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2319.119703] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2329.998926] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2330.775262] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2331.776368] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2333.905208] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2333.907461] Lustre: Skipped 1 previous similar message [ 2338.660138] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2338.662760] Lustre: Skipped 3 previous similar messages [ 2347.381783] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2347.385629] Lustre: Skipped 7 previous similar messages [ 2363.592139] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2363.601513] Lustre: Skipped 15 previous similar messages [ 2396.224569] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2396.228370] Lustre: Skipped 32 previous similar messages [ 2441.048947] Lustre: 87764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2444.724607] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:54:25 (1787615665) [ 2447.097033] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2449.943320] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:54:30 (1787615670) [ 2457.059771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2457.062459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.083225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2457.111886] Lustre: Skipped 1 previous similar message [ 2462.179418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2462.187585] Lustre: Skipped 2 previous similar messages [ 2467.307594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.323711] Lustre: Skipped 1 previous similar message [ 2469.347240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2469.783335] Lustre: server umount lustre-OST0000 complete [ 2474.977367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2474.989496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.001180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.537837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.261186] Lustre: server umount lustre-MDT0000 complete [ 2482.660160] LustreError: 79651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.685027] LustreError: 79651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2486.143810] LustreError: 82248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615709 with bad export cookie 12048203517646917513 [ 2486.154180] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2486.162579] LustreError: 82248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2486.819641] Lustre: server umount lustre-MDT0001 complete [ 2500.010928] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2503.660542] Key type lgssc unregistered [ 2503.949565] LNet: 89022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2503.957781] LNetError: 89022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2503.977753] LNet: Removed LNI 192.168.202.137@tcp [ 2505.049253] Key type .llcrypt unregistered [ 2505.051325] Key type ._llcrypt unregistered [ 2510.931258] Key type ._llcrypt registered [ 2510.932863] Key type .llcrypt registered [ 2511.056306] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2517.364650] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules [ 2518.924989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2518.971265] alg: No test for adler32 (adler32-zlib) [ 2520.012381] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 2520.373860] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2522.104541] Key type lgssc registered [ 2523.898430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2525.492794] LustreError: 90243:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2545.139794] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2547.913172] LustreError: 90243:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2557.416336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2557.450293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.377574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.294997] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2569.699796] LustreError: 91252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.748993] LustreError: 91252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2570.726045] LustreError: 91251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.847361] LustreError: 91252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.965770] LustreError: 91251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.990902] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:56:42 (1787615802) [ 2585.058300] LustreError: 91252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.818494] LustreError: 91283:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2585.832537] LustreError: 91283:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2585.839184] LustreError: 91283:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 26, retries 0, failed: rc = -5 [ 2586.101503] Lustre: server umount lustre-MDT0000 complete [ 2602.080958] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2605.235700] Key type lgssc unregistered [ 2605.731531] LNet: 92781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.739453] LNetError: 92781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.782738] LNet: Removed LNI 192.168.202.137@tcp [ 2606.749401] Key type .llcrypt unregistered [ 2606.754217] Key type ._llcrypt unregistered [ 2629.749483] Key type ._llcrypt registered [ 2629.752665] Key type .llcrypt registered [ 2629.908989] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2631.356699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2631.401382] alg: No test for adler32 (adler32-zlib) [ 2632.593800] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 2632.994764] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2634.737086] Key type lgssc registered [ 2636.290748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2646.225604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2646.242628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.922811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2652.803958] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2657.250260] LustreError: 94004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.271534] LustreError: 94004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2660.997996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.751984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2667.416454] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2678.478932] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2687.071056] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2696.058976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.941440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.377598] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2710.398476] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2772.483321] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:59:53 (1787615993) [ 2774.441457] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2776.518944] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:59:58 (1787615998) [ 2782.183505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.184948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.205660] Lustre: Skipped 2 previous similar messages [ 2782.206251] Lustre: Skipped 1 previous similar message [ 2787.166316] Lustre: server umount lustre-MDT0000 complete [ 2787.299803] LustreError: 93999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.307832] LustreError: 93999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2790.171880] LustreError: 93984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616013 with bad export cookie 9362631931752306694 [ 2790.196020] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2790.418609] Lustre: server umount lustre-MDT0001 complete [ 2804.142960] Lustre: server umount lustre-OST0000 complete [ 2816.877654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.710741] LustreError: 98316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.958840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.141797] LustreError: 98317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.235522] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2827.301332] LustreError: 98316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.149726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.824259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2840.206127] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2849.052739] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2854.983111] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2867.577808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2867.714435] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2867.718231] LustreError: 98306:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2867.732213] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2867.736636] LustreError: 100412:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2867.739930] LustreError: 100412:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2867.742977] LustreError: 100412:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2867.777236] Lustre: server umount lustre-OST0000 complete [ 2867.779400] LustreError: 100412:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2874.904920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.031104] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2875.039646] LustreError: 98306:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2875.046163] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2875.052208] LustreError: 100873:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2875.061951] LustreError: 100873:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2875.068063] LustreError: 100873:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2875.146578] Lustre: server umount lustre-OST0000 complete [ 2875.148669] LustreError: 100873:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2884.904448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2885.096491] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2885.102580] Lustre: Found index 0 for lustre-OST0000, updating log [ 2885.113768] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2885.265563] Lustre: lustre-OST0000: new disk, initializing [ 2885.268976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2885.275111] Lustre: 101410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2885.758774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.695359] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2893.644829] Lustre: Failing over lustre-OST0000 [ 2893.852314] Lustre: server umount lustre-OST0000 complete [ 2898.402411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.405726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2898.427808] Lustre: Skipped 1 previous similar message [ 2898.440944] Lustre: Skipped 2 previous similar messages [ 2903.526354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.537961] Lustre: Skipped 1 previous similar message [ 2909.665895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2909.671788] Lustre: Skipped 3 previous similar messages [ 2911.200211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2911.760196] Lustre: server umount lustre-MDT0000 complete [ 2914.786322] LustreError: 98311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.805933] LustreError: 98311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2916.314176] LustreError: 101409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616139 with bad export cookie 9362631931752309438 [ 2916.338562] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2916.341961] LustreError: 101409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2916.656717] Lustre: server umount lustre-MDT0001 complete [ 2926.201307] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:02:27 (1787616147) [ 2943.728299] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2955.824691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.278462] LustreError: 103772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.352695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.229231] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2969.922712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.760190] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2980.165357] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2987.439387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2988.597280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2988.608259] Lustre: Skipped 1 previous similar message [ 2988.617171] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2995.821603] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2995.832312] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2995.850129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2996.294878] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3000.967061] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 3001.006853] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 3001.016654] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3001.038581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3001.058904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3005.917621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3005.989467] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3006.069759] Lustre: lustre-OST0001: new disk, initializing [ 3006.073240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3006.077622] Lustre: 106091:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3012.714492] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3013.173812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3013.182659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3013.309040] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3020.887819] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3028.623509] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3034.093797] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.098510] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3034.115179] Lustre: Skipped 2 previous similar messages [ 3034.146843] Lustre: Skipped 2 previous similar messages [ 3038.398449] Lustre: server umount lustre-MDT0001 complete [ 3039.214062] LustreError: 103773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.252642] LustreError: 103773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3049.435556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.577252] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3049.582048] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3049.643859] Lustre: 103760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3051.599694] LustreError: 107637:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3051.618477] LustreError: 107637:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3051.658445] LustreError: 103764:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3051.807177] LustreError: 107648:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.137@tcp: cfg command failed: rc = -17 [ 3051.824129] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3051.824129] [ 3055.492716] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3056.119230] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3066.697386] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 20:04:48 (1787616288) [ 3084.212819] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3099.137266] Lustre: 109709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3121.771813] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3125.663782] Lustre: 110844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3213.799426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3213.811117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.838572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3213.845672] Lustre: Skipped 1 previous similar message [ 3216.966298] Lustre: server umount lustre-MDT0000 complete [ 3219.941043] LustreError: 103767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.963395] LustreError: 103767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3221.681700] LustreError: 103754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616444 with bad export cookie 9362631931752310369 [ 3221.684969] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3221.688748] LustreError: 103754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3221.965411] Lustre: server umount lustre-MDT0001 complete [ 3237.631450] Lustre: server umount lustre-OST0000 complete [ 3251.401371] Lustre: server umount lustre-OST0001 complete [ 3258.983035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3271.528453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3280.831577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3292.908583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3319.915457] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3332.657498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.684847] Lustre: lustre-MDT0000: reset Object Index mappings [ 3333.260907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.265286] Lustre: Skipped 2 previous similar messages [ 3338.217830] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3348.969711] LustreError: 114798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3348.997244] LustreError: 114798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3349.054134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3349.084987] Lustre: lustre-MDT0001: reset Object Index mappings [ 3355.203321] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3358.459503] Lustre: 115948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3364.975695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3373.021670] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3374.037386] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3378.172693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3382.008654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3387.900044] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3387.932252] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3390.662566] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3399.245788] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3402.722786] Lustre: 117824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3460.126690] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:11:21 (1787616681) [ 3485.173724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.186246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.198111] Lustre: Skipped 5 previous similar messages [ 3485.200615] Lustre: Skipped 5 previous similar messages [ 3491.424471] Lustre: server umount lustre-MDT0000 complete [ 3496.142975] LustreError: 114780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616719 with bad export cookie 9362631931752357486 [ 3496.150804] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3496.160280] LustreError: 114780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3500.515139] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.539205] Lustre: Skipped 2 previous similar messages [ 3502.727857] Lustre: server umount lustre-MDT0001 complete [ 3513.856298] Lustre: server umount lustre-OST0000 complete [ 3524.968766] Lustre: server umount lustre-OST0001 complete [ 3532.909331] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 3546.411397] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3590.598611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.866589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3590.904993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3591.043729] Lustre: lustre-MDT0000: new disk, initializing [ 3591.104485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.107035] Lustre: Skipped 3 previous similar messages [ 3591.128712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3595.776958] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3603.709623] Lustre: server umount lustre-MDT0000 complete [ 3614.618892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.095312] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3615.121622] Lustre: 122101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3615.141372] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3620.424249] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3625.929899] Lustre: server umount lustre-MDT0000 complete [ 3636.468774] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:14:17 (1787616857) [ 3638.575722] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3640.964889] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:14:22 (1787616862) [ 3660.785628] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3673.249256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3679.393732] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3689.783397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.943383] Lustre: 124152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3690.025968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3690.031194] Lustre: Skipped 1 previous similar message [ 3690.140450] Lustre: lustre-MDT0001: new disk, initializing [ 3690.381391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3690.397303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3695.540451] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3700.881737] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3708.511349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3708.937071] Lustre: lustre-OST0000: new disk, initializing [ 3708.951207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3708.961941] Lustre: 125729:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3710.582541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3710.597977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3710.782598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3717.464551] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3728.971496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3729.173975] Lustre: lustre-OST0001: new disk, initializing [ 3729.177543] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3729.182707] Lustre: 126601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3731.282755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3731.301946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3731.550866] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3737.126677] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3748.257110] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3752.221758] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4171.900988] Lustre: 133019:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4171.911847] Lustre: 133019:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4171.918712] Lustre: 133019:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4171.934386] Lustre: 133019:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4171.945779] Lustre: 133019:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4171.963778] Lustre: 133019:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4500.196123] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:28:40 (1787617720) [ 4514.787624] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.800545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4514.825247] Lustre: Skipped 2 previous similar messages [ 4514.843424] Lustre: Skipped 13 previous similar messages [ 4519.907020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4519.917700] Lustre: Skipped 2 previous similar messages [ 4521.151335] Lustre: server umount lustre-MDT0000 complete [ 4525.039282] LustreError: 127388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.067095] LustreError: 127388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 4525.213489] LustreError: 133032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617748 with bad export cookie 9362631931752426681 [ 4525.218564] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4525.226971] LustreError: 133032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4530.147496] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4530.165451] Lustre: Skipped 2 previous similar messages [ 4530.167235] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4530.179789] Lustre: Skipped 1 previous similar message [ 4532.104507] Lustre: server umount lustre-MDT0001 complete [ 4542.776553] Lustre: server umount lustre-OST0000 complete [ 4561.376229] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4561.752981] Lustre: server umount lustre-OST0001 complete [ 4569.289927] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4579.134380] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4623.463548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4623.743310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4623.776751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4623.857571] Lustre: lustre-MDT0000: new disk, initializing [ 4623.960959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4623.973588] Lustre: Skipped 5 previous similar messages [ 4623.987373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4629.771511] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4642.353319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.520996] Lustre: 136084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4642.633299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4642.645965] Lustre: Skipped 1 previous similar message [ 4642.725865] Lustre: lustre-MDT0001: new disk, initializing [ 4642.830746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4642.845610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4647.872504] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4658.524981] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4665.052886] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4672.879256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4673.176866] Lustre: lustre-OST0000: new disk, initializing [ 4673.182397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4673.199632] Lustre: 138051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4673.332603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4673.339175] Lustre: Skipped 1 previous similar message [ 4674.519294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4674.531835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4674.804853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4679.944662] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4691.759499] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4704.549107] Lustre: lustre-MDD0000: changelog on [ 4707.742965] Lustre: lustre-MDD0001: changelog on [ 6861.188195] Lustre: lustre-MDD0001: changelog off [ 6863.549913] Lustre: 136090:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6864.718794] Lustre: lustre-MDD0000: changelog off [ 6876.132235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6876.142209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6876.155132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6876.650128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6876.654074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6881.772988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6881.778604] Lustre: Skipped 2 previous similar messages [ 6886.368171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6886.759376] Lustre: server umount lustre-OST0000 complete [ 6895.589260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6895.600462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.629168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6897.636838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.216961] Lustre: server umount lustre-MDT0000 complete [ 6902.755266] LustreError: 136093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6902.776232] LustreError: 136093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6903.892656] LustreError: 136074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620127 with bad export cookie 9362631931753253178 [ 6903.895808] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6903.901042] LustreError: 136074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6904.242362] Lustre: server umount lustre-MDT0001 complete [ 6917.645677] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6920.143317] Key type lgssc unregistered [ 6920.391523] LNet: 147252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6920.400623] LNetError: 147252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6920.414865] LNet: Removed LNI 192.168.202.137@tcp [ 6921.244236] Key type .llcrypt unregistered [ 6921.246825] Key type ._llcrypt unregistered [ 6946.237866] Key type ._llcrypt registered [ 6946.241544] Key type .llcrypt registered [ 6946.382790] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 6971.746606] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6973.688867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6973.699793] alg: No test for adler32 (adler32-zlib) [ 6975.004413] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 6975.690106] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6977.576286] Key type lgssc registered [ 6978.780232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7018.261342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7018.302054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7019.722156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7019.809397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7019.962154] Lustre: lustre-MDT0000: new disk, initializing [ 7020.109130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7020.131937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7024.629170] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7039.403560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7039.649669] Lustre: 150460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7039.770750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7039.782175] Lustre: Skipped 1 previous similar message [ 7039.914720] Lustre: lustre-MDT0001: new disk, initializing [ 7040.052646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7040.090840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7040.114273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7045.538050] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7057.567089] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7065.552362] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7077.030898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7077.509968] Lustre: lustre-OST0000: new disk, initializing [ 7077.519756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7077.534055] Lustre: 152432:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7077.650931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.617547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7083.626823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7083.700301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7087.653542] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7101.868463] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7113.709094] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_module obdecho/obdecho [ 7116.717676] LustreError: 153437:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7116.737319] LustreError: 153437:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7119.397301] LustreError: 153533:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7135.222029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.237432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7135.238575] Lustre: Skipped 1 previous similar message [ 7135.254121] Lustre: Skipped 2 previous similar messages [ 7139.332910] Lustre: server umount lustre-MDT0000 complete [ 7145.442085] LustreError: 150470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.468437] LustreError: 150470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7145.855992] LustreError: 150450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620368 with bad export cookie 17081933040865309850 [ 7145.864085] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7145.870736] LustreError: 150450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7146.488111] Lustre: server umount lustre-MDT0001 complete [ 7163.263372] Lustre: server umount lustre-OST0000 complete [ 7190.367221] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 7208.182288] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7255.504606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.807236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7255.866752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7256.047381] Lustre: lustre-MDT0000: new disk, initializing [ 7256.198218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.221744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7260.626818] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7276.025361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.270887] Lustre: 157276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7276.370324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7276.374781] Lustre: Skipped 1 previous similar message [ 7276.503335] Lustre: lustre-MDT0001: new disk, initializing [ 7276.674090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7276.817227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7276.864226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7284.756028] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7295.678573] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7302.438089] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7310.469250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7310.781363] Lustre: lustre-OST0000: new disk, initializing [ 7310.785694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7310.791037] Lustre: 159244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7310.872626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.927758] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7318.123036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7318.133640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7318.334445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7330.965838] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7337.939589] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7338.131826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7343.986969] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7344.164272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7349.224843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7349.231244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7349.236427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7349.242896] Lustre: Skipped 2 previous similar messages [ 7349.252403] Lustre: Skipped 1 previous similar message [ 7352.547272] Lustre: server umount lustre-OST0000 complete [ 7358.434245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7358.437653] LustreError: Skipped 1 previous similar message [ 7358.442482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7358.452396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7364.006664] Lustre: server umount lustre-MDT0000 complete [ 7364.577595] LustreError: 157279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7364.602136] LustreError: 157279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7368.185789] LustreError: 157267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620591 with bad export cookie 17081933040865310809 [ 7368.190524] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7368.197640] LustreError: 157267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7368.580327] Lustre: server umount lustre-MDT0001 complete [ 7372.747876] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:16:34 (1787620594) [ 7374.355899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7376.506909] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:16:37 (1787620597) [ 7384.794924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.333255] LustreError: 161657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7385.379538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.404877] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7390.690780] LustreError: 161658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.814128] LustreError: 161657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7401.741044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7408.115882] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7418.681340] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7426.176303] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7433.392413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7433.656302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7433.661666] Lustre: Skipped 1 previous similar message [ 7443.230188] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7456.813977] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7465.443324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7465.451436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7465.486950] Lustre: Skipped 3 previous similar messages [ 7465.504050] Lustre: Skipped 3 previous similar messages [ 7469.599988] Lustre: server umount lustre-MDT0000 complete [ 7470.561806] LustreError: 161658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7470.585780] LustreError: 161658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7473.780200] LustreError: 161637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620696 with bad export cookie 17081933040865311691 [ 7473.790207] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7473.799892] LustreError: 161637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7475.683503] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.709173] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7475.722410] Lustre: Skipped 2 previous similar messages [ 7480.129719] Lustre: server umount lustre-MDT0001 complete [ 7488.401219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7502.523586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7502.986509] LustreError: 165501:0:(llog.c:1655:llog_backup()) MGC192.168.202.137@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7503.015818] Lustre: 165501:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.137@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7503.336466] LustreError: 165506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7503.350948] LustreError: 165506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7539.174988] LustreError: 165526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.212620] LustreError: 165526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7543.354046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7543.363475] Lustre: Skipped 2 previous similar messages [ 7563.744501] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7563.746375] Lustre: 165501:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7604.717985] LustreError: 165508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7604.719029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7604.741344] LustreError: 165508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7609.829332] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7610.886441] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7621.755990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.389401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7643.658746] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xed0f3798c1c91508 [ 7643.695649] Lustre: MGC192.168.202.137@tcp: Connection restored to 0@lo (at 0@lo) [ 7649.264353] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7650.229584] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7661.052231] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7663.535543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7670.748294] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7672.760428] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7700.460946] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7700.472902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7700.477066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7700.866453] Lustre: server umount lustre-MDT0000 complete [ 7708.935298] LustreError: 164442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620932 with bad export cookie 17081933040865318152 [ 7708.949815] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7708.959974] LustreError: 164442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7710.224230] Lustre: server umount lustre-MDT0001 complete [ 7728.124338] Lustre: server umount lustre-OST0000 complete [ 7755.792058] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 7769.849686] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7816.546754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.717583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7816.757544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7816.804736] Lustre: lustre-MDT0000: new disk, initializing [ 7816.845579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.851145] Lustre: Skipped 1 previous similar message [ 7816.858660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7821.383959] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7835.553310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.890878] Lustre: 170631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7836.209103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7842.764414] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7855.425957] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7864.380106] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7874.745641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7875.225832] Lustre: lustre-OST0000: new disk, initializing [ 7875.231592] Lustre: Skipped 1 previous similar message [ 7875.245191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7875.252323] Lustre: Skipped 2 previous similar messages [ 7875.263937] Lustre: 172601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7883.569934] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7884.912030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7884.923427] Lustre: Skipped 1 previous similar message [ 7884.925602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7885.027584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7892.395082] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7896.878962] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7897.043272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7902.408842] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7902.630126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7905.767976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7905.770384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7905.778425] LustreError: Skipped 1 previous similar message [ 7905.779161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7905.788289] Lustre: Skipped 3 previous similar messages [ 7905.865578] Lustre: Skipped 3 previous similar messages [ 7910.895311] LustreError: 172585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7910.919574] LustreError: 172585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 36 previous similar messages [ 7911.014056] Lustre: server umount lustre-OST0000 complete [ 7930.336827] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7930.783611] Lustre: server umount lustre-MDT0000 complete [ 7935.223039] LustreError: 170621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621158 with bad export cookie 17081933040865334980 [ 7935.225683] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7935.233597] LustreError: 170621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7935.779587] Lustre: server umount lustre-MDT0001 complete [ 7945.783269] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:26:07 (1787621167) [ 7952.969847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7959.220770] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7970.193801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.494846] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7985.329285] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7993.900536] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8004.093583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8013.356636] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8022.560603] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8033.615671] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:27:34 (1787621254) [ 8038.884170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8038.891615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8038.906304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8038.913817] Lustre: Skipped 3 previous similar messages [ 8038.930617] Lustre: Skipped 9 previous similar messages [ 8044.714652] Lustre: server umount lustre-OST0000 complete [ 8055.352920] Lustre: server umount lustre-MDT0000 complete [ 8059.630359] LustreError: 174994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621282 with bad export cookie 17081933040865335862 [ 8059.644431] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8060.112890] Lustre: server umount lustre-MDT0001 complete [ 8071.228698] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8074.109989] Key type lgssc unregistered [ 8074.375887] LNet: 178747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8074.381140] LNetError: 178747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8074.400975] LNet: Removed LNI 192.168.202.137@tcp [ 8075.257313] Key type .llcrypt unregistered [ 8075.260906] Key type ._llcrypt unregistered [ 8096.153317] Key type ._llcrypt registered [ 8096.156687] Key type .llcrypt registered [ 8096.297911] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8097.565705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8097.743499] alg: No test for adler32 (adler32-zlib) [ 8098.749383] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 8098.945219] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8100.648205] Key type lgssc registered [ 8102.276308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8111.312633] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8111.382696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8127.136504] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8132.256603] LustreError: 179991:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 8137.376875] LustreError: 179991:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8137.391619] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8137.399171] LustreError: MGC192.168.202.137@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8137.412210] LustreError: 179991:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8137.422174] LustreError: 179991:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 8137.430744] LustreError: 179991:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8137.505415] Lustre: server umount lustre-OST0000 complete [ 8137.508625] LustreError: 179991:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8147.548415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8149.097620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8155.139930] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8159.714183] LustreError: 180491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8159.745581] LustreError: 180491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8164.837828] LustreError: 180490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8166.058962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8166.438264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8172.070783] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8181.126317] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8188.071243] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8195.671483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8195.911029] LustreError: 182308:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8195.919542] Lustre: 182308:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.137@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8196.109640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8203.212544] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8211.531864] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8216.553172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8216.564911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8216.569317] Lustre: Skipped 2 previous similar messages [ 8216.583357] Lustre: Skipped 2 previous similar messages [ 8221.671921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8221.679592] Lustre: Skipped 2 previous similar messages [ 8226.790283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8226.799569] Lustre: Skipped 2 previous similar messages [ 8228.320635] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8228.516125] Lustre: server umount lustre-MDT0000 complete [ 8231.906693] LustreError: 180491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.930748] LustreError: 180491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8232.679538] LustreError: 180470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621455 with bad export cookie 13732353517009362377 [ 8232.692227] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8232.709039] LustreError: 180470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8233.127207] Lustre: server umount lustre-MDT0001 complete [ 8247.096558] Lustre: server umount lustre-OST0000 complete [ 8258.427414] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 8269.016238] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8313.315396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8313.867969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8313.935976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8314.266799] Lustre: lustre-MDT0000: new disk, initializing [ 8314.393681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8314.417141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8318.846269] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8335.142291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8335.320414] Lustre: 185835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8335.386761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8335.390996] Lustre: Skipped 1 previous similar message [ 8335.549159] Lustre: lustre-MDT0001: new disk, initializing [ 8335.684689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8335.721898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8335.733163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8341.834230] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8355.393389] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8363.471023] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8373.392558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8373.730213] Lustre: lustre-OST0000: new disk, initializing [ 8373.736168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8373.742156] Lustre: 187808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8373.832927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8380.554455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8380.570177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8380.741414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8382.429045] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8392.549987] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8398.337982] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8398.658254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8404.271175] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8404.458658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8406.497642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8406.502445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8406.509551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8406.514812] Lustre: Skipped 1 previous similar message [ 8406.529213] Lustre: Skipped 1 previous similar message [ 8411.618793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8411.626462] Lustre: Skipped 1 previous similar message [ 8412.739658] Lustre: server umount lustre-OST0000 complete [ 8417.763611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8417.770318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8421.868342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8421.888455] Lustre: Skipped 1 previous similar message [ 8421.890381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.549673] Lustre: server umount lustre-MDT0000 complete [ 8426.984960] LustreError: 185844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8427.034202] LustreError: 185844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8428.202113] LustreError: 185828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621651 with bad export cookie 13732353517009363098 [ 8428.227894] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8428.230539] LustreError: 185828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8428.675400] Lustre: server umount lustre-MDT0001 complete [ 8442.867237] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:34:23 (1787621663) [ 8464.849249] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8468.164703] Key type lgssc unregistered [ 8468.476859] LNet: 190757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8468.503306] LNetError: 190757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8469.544550] LNet: Removed LNI 192.168.202.137@tcp [ 8470.581768] Key type .llcrypt unregistered [ 8470.584226] Key type ._llcrypt unregistered [ 8492.663350] Key type ._llcrypt registered [ 8492.675173] Key type .llcrypt registered [ 8492.842690] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8493.992114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8494.005930] alg: No test for adler32 (adler32-zlib) [ 8495.001905] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 8495.432788] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8497.400231] Key type lgssc registered [ 8499.184098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8509.621199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8509.638566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8511.257404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8516.834502] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8521.700673] LustreError: 191976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.720970] LustreError: 191976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8526.818432] LustreError: 191975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.511674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8528.030952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8533.688481] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8543.547558] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8550.542844] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8559.404484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8559.658882] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8559.660278] LustreError: 193838:0:(llog.c:1696:llog_backup()) MGC192.168.202.137@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8559.662820] Lustre: 193838:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.137@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8559.762631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8567.602935] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8577.029981] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8583.649551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8583.652452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8583.667731] Lustre: Skipped 2 previous similar messages [ 8583.689609] Lustre: Skipped 2 previous similar messages [ 8588.774067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8588.780687] Lustre: Skipped 2 previous similar messages [ 8593.892262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8593.915132] Lustre: Skipped 2 previous similar messages [ 8594.400159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8594.661151] Lustre: server umount lustre-MDT0000 complete [ 8598.515257] LustreError: 191956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621821 with bad export cookie 9766674545643266888 [ 8598.535410] LustreError: 191956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8598.536545] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8598.768545] Lustre: server umount lustre-MDT0001 complete [ 8614.005839] Lustre: server umount lustre-OST0000 complete [ 8625.263385] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 8636.956216] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8683.234323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8683.549101] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8683.582466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8683.732969] Lustre: lustre-MDT0000: new disk, initializing [ 8683.932858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8684.004466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8688.881648] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8702.194514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.412459] Lustre: 197368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8702.469811] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8702.473107] Lustre: Skipped 1 previous similar message [ 8702.476824] LustreError: 198153:0:(llog.c:1696:llog_backup()) MGC192.168.202.137@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8702.486627] LustreError: 198153:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8702.492660] Lustre: 198153:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.137@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8702.580218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8702.582395] Lustre: Skipped 1 previous similar message [ 8702.798621] Lustre: lustre-MDT0001: new disk, initializing [ 8703.002894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8703.089082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8703.109282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8707.460459] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8720.275879] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8727.837987] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8736.548119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8736.844909] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8736.846688] Lustre: Skipped 1 previous similar message [ 8736.850360] LustreError: 199315:0:(llog.c:1696:llog_backup()) MGC192.168.202.137@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8736.862749] LustreError: 199315:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8736.866775] Lustre: 199315:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.137@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8736.877515] Lustre: 199315:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8736.901734] Lustre: lustre-OST0000: new disk, initializing [ 8736.907602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8736.913081] Lustre: 199340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8737.008334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8742.578801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8742.594233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8742.698886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8744.558137] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8756.351570] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8761.972583] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8762.238753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8767.754544] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8768.084075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8773.606502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8773.615380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.620365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8773.633253] Lustre: Skipped 1 previous similar message [ 8773.654760] Lustre: Skipped 1 previous similar message [ 8776.982638] Lustre: server umount lustre-OST0000 complete [ 8783.849989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8783.859245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8783.859635] Lustre: Skipped 1 previous similar message [ 8783.867598] Lustre: Skipped 1 previous similar message [ 8788.496291] Lustre: server umount lustre-MDT0000 complete [ 8788.969897] LustreError: 197380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8788.995474] LustreError: 197380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8793.058800] LustreError: 197380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8793.083595] LustreError: 197380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8793.486640] LustreError: 197361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622016 with bad export cookie 9766674545643267602 [ 8793.488859] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8793.508451] LustreError: 197361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8793.852410] Lustre: server umount lustre-MDT0001 complete [ 8804.296989] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:40:25 (1787622025) [ 8823.408733] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8835.545054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8835.988585] LustreError: 202468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8836.085081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8842.230677] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8846.308359] LustreError: 202469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.330365] LustreError: 202469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8851.453719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8857.538611] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8861.013891] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8868.211762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8868.534229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8868.541927] Lustre: Skipped 1 previous similar message [ 8874.440307] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8883.674419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8883.966737] Lustre: lustre-OST0001: new disk, initializing [ 8883.979964] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8883.991770] Lustre: 204774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8889.408147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8889.420132] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8889.493355] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8890.311499] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8903.117275] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8912.314746] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8915.339628] Lustre: DEBUG MARKER: ADD OST3 [ 8920.654172] loop: module loaded [ 8920.758426] loop0: detected capacity change from 0 to 204800000 [ 8921.101853] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8921.111357] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8921.118578] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8927.811546] loop0: detected capacity change from 0 to 204800000 [ 8928.889766] Lustre: DEBUG MARKER: STOP OST3 [ 8936.202324] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8936.477644] Lustre: lustre-OST0003: new disk, initializing [ 8936.489368] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8936.502851] Lustre: 206771:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8936.719184] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8936.738391] Lustre: Skipped 1 previous similar message [ 8938.511099] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8938.734339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8938.747686] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8938.788064] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8940.578822] LustreError: 206933:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8941.690126] Lustre: Failing over lustre-OST0003 [ 8942.112449] Lustre: server umount lustre-OST0003 complete [ 8943.154938] LustreError: 203949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8943.185674] LustreError: 203949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8944.117714] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8944.136067] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.160196] Lustre: Skipped 1 previous similar message [ 8945.045367] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8955.941249] Lustre: DEBUG MARKER: START OST3 again [ 8964.914259] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8965.380817] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8966.899667] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8967.194975] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8967.201645] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8974.109781] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8997.050530] Lustre: Failing over lustre-OST0003 [ 8997.203280] Lustre: server umount lustre-OST0003 complete [ 8997.865880] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8997.878745] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8997.895185] LustreError: 204781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8997.922504] LustreError: 204781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 9016.808862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9016.819391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9016.833843] Lustre: Skipped 4 previous similar messages [ 9016.847033] Lustre: Skipped 3 previous similar messages [ 9022.551234] Lustre: server umount lustre-MDT0000 complete [ 9026.525324] LustreError: 202448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622249 with bad export cookie 9766674545643268484 [ 9026.553654] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9033.082653] Lustre: server umount lustre-MDT0001 complete [ 9038.159338] Lustre: server umount lustre-OST0000 complete [ 9042.415519] Lustre: server umount lustre-OST0001 complete [ 9049.474648] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9060.554295] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9104.406243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9104.868717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9104.986631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9105.214909] Lustre: lustre-MDT0000: new disk, initializing [ 9105.460921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9105.471526] Lustre: Skipped 1 previous similar message [ 9105.516114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9112.464345] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9125.811676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9125.968504] Lustre: 212244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9126.066808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9126.074958] Lustre: Skipped 1 previous similar message [ 9126.166246] Lustre: lustre-MDT0001: new disk, initializing [ 9126.326850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9126.342804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9131.124871] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9143.416519] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9152.393990] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9163.743245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9164.087392] Lustre: lustre-OST0000: new disk, initializing [ 9164.091795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9164.106859] Lustre: 214216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9165.776674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9165.784541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9165.903596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9171.323056] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9182.658390] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9188.184117] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9188.458736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9193.883918] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9194.180742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9201.634456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9201.641403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9201.649236] Lustre: Skipped 2 previous similar messages [ 9201.662041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9201.667447] LustreError: Skipped 1 previous similar message [ 9201.697531] Lustre: Skipped 9 previous similar messages [ 9211.360152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9211.796541] Lustre: server umount lustre-OST0000 complete [ 9222.925468] Lustre: server umount lustre-MDT0000 complete [ 9227.257638] LustreError: 214887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9227.284680] LustreError: 214887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 9227.536697] LustreError: 212234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622450 with bad export cookie 9766674545643270780 [ 9227.561728] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9227.563216] LustreError: 212234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9228.289666] Lustre: server umount lustre-MDT0001 complete [ 9233.453631] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:47:34 (1787622454) [ 9253.431456] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9265.305233] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9311.276190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9311.463773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9311.529223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9311.676817] Lustre: lustre-MDT0000: new disk, initializing [ 9311.763240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9316.454437] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9332.353055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9332.574451] Lustre: 218831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9333.061323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9341.554964] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9352.578678] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9361.430622] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9369.837932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9370.243307] Lustre: 220804:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9370.401488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9370.411151] Lustre: Skipped 4 previous similar messages [ 9372.170207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9372.416973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9378.242056] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9393.038862] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9403.197140] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9403.456422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9410.116681] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9410.424461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9413.604868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9413.615278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9413.621391] Lustre: Skipped 3 previous similar messages [ 9413.625535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9413.627828] Lustre: Skipped 7 previous similar messages [ 9419.225493] Lustre: server umount lustre-OST0000 complete [ 9429.803163] Lustre: server umount lustre-MDT0000 complete [ 9434.084928] LustreError: 221766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9434.115037] LustreError: 221766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9434.263743] LustreError: 218823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622657 with bad export cookie 9766674545643271662 [ 9434.268948] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9434.271362] LustreError: 218823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9434.982490] Lustre: server umount lustre-MDT0001 complete [ 9447.959530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.023975] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9466.107882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9473.928074] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9483.026798] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9490.791694] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9502.494587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9510.884306] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9520.571504] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9531.886868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9531.909077] LustreError: Skipped 1 previous similar message [ 9531.919293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9531.944178] Lustre: Skipped 3 previous similar messages [ 9531.954603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9531.963158] Lustre: Skipped 6 previous similar messages [ 9537.918808] Lustre: server umount lustre-OST0000 complete [ 9549.171597] Lustre: server umount lustre-MDT0000 complete [ 9553.378110] LustreError: 223006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622776 with bad export cookie 9766674545643272544 [ 9553.385353] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9553.389295] LustreError: 223006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9553.950380] Lustre: server umount lustre-MDT0001 complete [ 9663.783036] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 9666.708190] Key type lgssc unregistered [ 9667.173194] LNet: 226696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9667.184853] LNetError: 226696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9667.201174] LNet: Removed LNI 192.168.202.137@tcp [ 9668.042252] Key type .llcrypt unregistered [ 9668.044743] Key type ._llcrypt unregistered [ 9703.691194] Key type ._llcrypt registered [ 9703.693140] Key type .llcrypt registered [ 9703.943849] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9726.676675] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9728.131867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9728.233389] alg: No test for adler32 (adler32-zlib) [ 9729.520332] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 9729.866979] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 9731.560185] Key type lgssc registered [ 9732.982447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9785.160345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9785.198568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9786.503253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9786.527554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9786.582691] Lustre: lustre-MDT0000: new disk, initializing [ 9786.727496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9786.796610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9792.452377] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9808.624751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9808.925705] Lustre: 230053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9809.037538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9809.055346] Lustre: Skipped 1 previous similar message [ 9809.289248] Lustre: lustre-MDT0001: new disk, initializing [ 9809.487572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9809.553435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9809.557023] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9816.897158] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9829.211714] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9835.930030] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9844.489619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9844.966089] Lustre: lustre-OST0000: new disk, initializing [ 9844.977646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9844.991586] Lustre: 232025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9845.178852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9852.876677] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9853.033954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9853.054987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9853.286816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9865.805384] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9871.652633] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9871.848191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9877.241934] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9877.556624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9884.130996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9884.148098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9884.154514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9884.162924] Lustre: Skipped 1 previous similar message [ 9886.938740] Lustre: server umount lustre-OST0000 complete [ 9894.372146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9894.401305] Lustre: Skipped 1 previous similar message [ 9894.436777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9894.446127] Lustre: Skipped 1 previous similar message [ 9898.563815] Lustre: server umount lustre-MDT0000 complete [ 9903.058053] LustreError: 230048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623126 with bad export cookie 15727670888773799838 [ 9903.060399] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9903.085736] LustreError: 230048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9903.625296] Lustre: server umount lustre-MDT0001 complete [ 9925.880551] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9936.708887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9937.067022] LustreError: 234962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9937.084180] LustreError: 234962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9937.114517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9941.412529] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9942.508173] LustreError: 234963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.620385] LustreError: 234962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.738517] LustreError: 234963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9954.303890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9955.527216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9963.210904] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9967.397236] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9975.637272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9976.364389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9985.136667] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9997.376300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9997.604760] Lustre: lustre-OST0001: new disk, initializing [ 9997.608814] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9997.622838] Lustre: 237276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9997.759036] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10003.094975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10003.125697] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10003.346208] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10006.867924] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10019.380865] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10029.933091] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10046.113857] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 22:01:07 (1787623267) [10049.507539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.522641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10049.539954] Lustre: Skipped 3 previous similar messages [10049.568228] Lustre: Skipped 3 previous similar messages [10054.638121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10057.751655] Lustre: server umount lustre-MDT0000 complete [10059.761735] LustreError: 235884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10059.814046] LustreError: 235884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10062.895678] LustreError: 235698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623286 with bad export cookie 15727670888773800720 [10062.903692] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10062.912213] LustreError: 235698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10063.255277] Lustre: server umount lustre-MDT0001 complete [10078.890694] Lustre: server umount lustre-OST0000 complete [10081.126842] Lustre: 228279:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787623288/real 1787623288] req@ffff8cbc8ece1500 x1874458524642304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787623304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10081.173088] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10081.185258] Lustre: Skipped 1 previous similar message [10084.008397] Lustre: server umount lustre-OST0001 complete [10091.339964] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [10102.387610] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10151.494227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10151.787890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10151.832730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10151.932519] Lustre: lustre-MDT0000: new disk, initializing [10152.009171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10152.052961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10157.517942] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10168.860746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.985265] Lustre: 241235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10169.034691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10169.038426] Lustre: Skipped 1 previous similar message [10169.140587] Lustre: lustre-MDT0001: new disk, initializing [10169.291037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10169.313386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10174.450825] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10187.233346] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10197.147779] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10204.164432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10204.596303] Lustre: lustre-OST0000: new disk, initializing [10204.603670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10204.615727] Lustre: 243209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10205.998714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10206.007236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10206.157138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10211.787265] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10222.477457] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10231.716544] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10232.166264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10239.411579] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10239.860958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10247.139067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10247.153457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10247.157246] LustreError: Skipped 1 previous similar message [10247.164823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10247.176112] Lustre: Skipped 1 previous similar message [10247.186763] Lustre: Skipped 3 previous similar messages [10249.852716] Lustre: server umount lustre-OST0000 complete [10257.381202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10257.386527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10257.414856] Lustre: Skipped 1 previous similar message [10257.439177] Lustre: Skipped 1 previous similar message [10262.810849] Lustre: server umount lustre-MDT0000 complete [10267.620439] LustreError: 242116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10267.663801] LustreError: 242116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10268.903567] LustreError: 241229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623492 with bad export cookie 15727670888773802855 [10268.907656] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10268.935207] LustreError: 241229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10269.742612] Lustre: server umount lustre-MDT0001 complete [10279.741045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10280.223763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10280.231091] Lustre: Skipped 2 previous similar messages [10285.553453] LustreError: 245430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.575600] LustreError: 245430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10289.005839] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10302.667924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10309.692760] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10321.748325] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10328.918123] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10336.926972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10344.388501] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10352.449776] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10503.650982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10503.659927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10503.694639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10503.700323] Lustre: Skipped 2 previous similar messages [10515.937042] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10516.235585] Lustre: server umount lustre-OST0000 complete [10523.625667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10523.648502] LustreError: Skipped 1 previous similar message [10527.029740] Lustre: server umount lustre-MDT0000 complete [10530.276368] LustreError: 247938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.308685] LustreError: 247938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10533.276182] LustreError: 245410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623756 with bad export cookie 15727670888773803730 [10533.305621] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10533.310470] LustreError: 245410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10533.906534] Lustre: server umount lustre-MDT0001 complete [10623.410681] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10627.815218] Key type lgssc unregistered [10628.112931] LNet: 249045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10628.126663] LNetError: 249045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10628.139239] LNet: Removed LNI 192.168.202.137@tcp [10629.429421] Key type .llcrypt unregistered [10629.433945] Key type ._llcrypt unregistered [10668.154822] Key type ._llcrypt registered [10668.157249] Key type .llcrypt registered [10668.294172] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [10692.477712] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10694.466337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10694.536256] alg: No test for adler32 (adler32-zlib) [10695.529766] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [10695.772439] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10697.416159] Key type lgssc registered [10698.897315] Lustre: Echo OBD driver; http://www.lustre.org/ [10749.045887] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10766.757272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10766.796603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10768.612454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10768.762425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10769.189515] Lustre: lustre-MDT0000: new disk, initializing [10769.553540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10769.609247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10783.595849] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10804.593681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10804.866335] Lustre: 253121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10804.961894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10804.968627] Lustre: Skipped 1 previous similar message [10805.134411] Lustre: lustre-MDT0001: new disk, initializing [10805.371855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10805.543259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10805.559901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10812.433336] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10818.854762] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10827.612439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10828.009692] Lustre: lustre-OST0000: new disk, initializing [10828.012505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10828.016919] Lustre: 254757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10828.111232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10828.363964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10828.378642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10828.624656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10837.373353] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10852.869222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10853.346252] Lustre: lustre-OST0001: new disk, initializing [10853.353637] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10853.369353] Lustre: 255630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10853.576363] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10862.395210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10862.403936] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10862.430512] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10862.520637] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10876.362824] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10883.890882] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10902.499226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10902.510428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10902.529533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10903.524911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10903.532778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10903.562489] Lustre: Skipped 2 previous similar messages [10904.926498] Lustre: server umount lustre-MDT0000 complete [10909.727256] LustreError: 253113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624132 with bad export cookie 10758078393336458499 [10909.728629] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10909.750114] LustreError: 253113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10910.205911] Lustre: server umount lustre-MDT0001 complete [10925.800262] Lustre: server umount lustre-OST0000 complete [10929.612798] Lustre: 250624:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787624136/real 1787624136] req@ffff8cbc822c4380 x1874459537527680/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787624152 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10940.122494] Lustre: server umount lustre-OST0001 complete [10956.491406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10968.923575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10977.910533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10988.729573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11009.074810] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11019.915345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11019.937343] Lustre: lustre-MDT0000: reset Object Index mappings [11020.349795] LustreError: 260644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11020.363554] LustreError: 260644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11020.435235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11025.903170] LustreError: 260645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11026.016721] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11029.993235] LustreError: 260644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11035.112256] LustreError: 260645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11035.809125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11035.850965] Lustre: lustre-MDT0001: reset Object Index mappings [11036.278858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11041.973137] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11045.344425] Lustre: 261789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11053.572867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11054.113896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11061.300301] LustreError: 262142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11061.428746] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11067.380248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [11070.939134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11076.601287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [11079.195671] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11089.012769] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11093.545041] Lustre: 263659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11100.376814] Lustre: server umount lustre-MDT0000 complete [11102.190656] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11102.195190] LustreError: 260644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.206356] Lustre: Skipped 3 previous similar messages [11102.245527] LustreError: 260644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11104.593787] LustreError: 260624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624327 with bad export cookie 10758078393336465296 [11104.603879] LustreError: 260624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11104.604978] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11104.909157] Lustre: server umount lustre-MDT0001 complete [11120.644257] Lustre: server umount lustre-OST0000 complete [11123.683266] Lustre: 250627:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787624330/real 1787624330] req@ffff8cbc8c369c00 x1874459537580160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787624346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11123.740110] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11125.537957] Lustre: 250625:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787624332/real 1787624332] req@ffff8cbd8ea98e00 x1874459537580416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787624348 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11125.775731] Lustre: server umount lustre-OST0001 complete [11155.939808] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 22:19:36 (1787624376) [11178.380556] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [11191.145461] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11239.002112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11239.198657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11239.214743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11239.290962] Lustre: lustre-MDT0000: new disk, initializing [11239.472418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11239.480859] Lustre: Skipped 1 previous similar message [11239.503096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11247.252743] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11261.769673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11261.907461] Lustre: 267918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11261.986759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11261.990534] Lustre: Skipped 1 previous similar message [11262.071239] Lustre: lustre-MDT0001: new disk, initializing [11262.229263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11262.241816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11269.216319] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11283.256851] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11292.274661] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11301.957768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11302.233311] Lustre: lustre-OST0000: new disk, initializing [11302.239438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11302.246726] Lustre: 269891:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11303.701039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11303.711051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11303.871637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11311.886503] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11326.137686] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11332.108987] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11332.428580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11338.020150] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11338.256322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11344.867428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11344.867781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11344.870166] LustreError: Skipped 1 previous similar message [11344.880077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11344.884792] Lustre: Skipped 4 previous similar messages [11347.375208] Lustre: server umount lustre-OST0000 complete [11354.598558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11354.623554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11354.643280] Lustre: Skipped 1 previous similar message [11354.662929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11360.234069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11360.256563] Lustre: Skipped 3 previous similar messages [11361.070943] Lustre: server umount lustre-MDT0000 complete [11365.347671] LustreError: 268786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11365.365229] LustreError: 268786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11366.277873] LustreError: 267910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624589 with bad export cookie 10758078393336467389 [11366.280969] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11366.284915] LustreError: 267910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11366.574379] Lustre: server umount lustre-MDT0001 complete [11391.153929] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11404.932218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11405.267541] LustreError: 272842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11405.301484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11405.307284] Lustre: Skipped 2 previous similar messages [11410.258561] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11419.148587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11423.903435] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11426.855538] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11434.268037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11441.038168] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11449.183654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11449.305797] Lustre: lustre-OST0001: new disk, initializing [11449.310423] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11449.321518] Lustre: 275151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11450.831870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11450.846219] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11451.087122] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11454.968554] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11467.600896] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11474.663398] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11481.537863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11481.815308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11493.349302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11493.360381] Lustre: Skipped 3 previous similar messages [11493.365403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11493.408358] Lustre: Skipped 3 previous similar messages [11494.512772] Lustre: server umount lustre-MDT0000 complete [11498.468927] LustreError: 272837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11498.512120] LustreError: 272837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11498.933643] LustreError: 272824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624722 with bad export cookie 10758078393336468271 [11498.934762] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11498.945539] LustreError: 272824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11499.400628] Lustre: server umount lustre-MDT0001 complete [11514.112863] Lustre: server umount lustre-OST0000 complete [11528.317564] Lustre: server umount lustre-OST0001 complete [11547.975798] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11558.023558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11558.397104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11558.405355] Lustre: Skipped 3 previous similar messages [11563.272941] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11572.349979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11578.156211] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11581.263995] Lustre: 279349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11588.471357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11595.701752] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11603.524754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11605.757882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11611.210408] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11613.447435] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11622.083960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11626.277950] Lustre: 281228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11632.045285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11632.081083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11644.307774] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:27:45 (1787624865) [11659.748043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11659.760987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11659.780914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11665.477702] Lustre: server umount lustre-MDT0000 complete [11667.428121] LustreError: 278202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.441574] LustreError: 278202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11669.373291] LustreError: 279726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624892 with bad export cookie 10758078393336474837 [11669.376566] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11669.380739] LustreError: 279726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11675.711416] Lustre: server umount lustre-MDT0001 complete [11686.288801] Lustre: server umount lustre-OST0000 complete [11690.857305] Lustre: server umount lustre-OST0001 complete [11698.220507] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [11709.082066] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11761.024411] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11774.168731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11774.394609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11774.445063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11774.521867] Lustre: lustre-MDT0000: new disk, initializing [11774.587924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11779.860424] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11791.401335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11791.611897] Lustre: 285472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11792.011451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11797.044990] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11811.629131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11812.079880] Lustre: 287105:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11813.562793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11821.648589] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11834.556889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11834.838818] Lustre: 287977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11834.985487] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11835.001061] Lustre: Skipped 6 previous similar messages [11836.744520] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11836.749711] Lustre: Skipped 1 previous similar message [11836.871692] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11841.402968] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11851.116226] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11855.122422] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11855.132317] Lustre: Skipped 1 previous similar message [11871.559604] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 22:31:33 (1787625093) [11876.357527] Lustre: server umount lustre-MDT0000 complete [11877.877123] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11877.905821] Lustre: Skipped 6 previous similar messages [11880.437437] LustreError: 285465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625103 with bad export cookie 10758078393336482495 [11880.449873] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11880.452786] LustreError: 285465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11881.078312] Lustre: server umount lustre-MDT0001 complete [11895.711639] Lustre: server umount lustre-OST0000 complete [11899.680921] Lustre: 250625:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787625106/real 1787625106] req@ffff8cbc84b86680 x1874459537854592/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787625122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11899.997984] Lustre: server umount lustre-OST0001 complete [11909.384873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11915.109318] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11923.420283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11930.350560] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11940.133071] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11946.843209] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11954.746208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11956.327212] LustreError: 292054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11956.354893] LustreError: 292054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [11963.092421] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11972.063713] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11989.429108] loop1: detected capacity change from 0 to 204800000 [11989.864593] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11989.870996] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11989.879704] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11996.688959] loop1: detected capacity change from 0 to 204800000 [12003.248222] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12003.313253] LustreError: 290210:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs21988: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [12003.326895] LustreError: 290210:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs21988 [12003.329410] LustreError: 290210:0:(mgs_handler.c:517:mgs_target_reg()) fs21988-MDT0001: Error checking target index: rc = -13 [12003.365190] LustreError: fs21988-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12003.381612] LustreError: 293947:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [12003.393829] LustreError: 293947:0:(tgt_mount.c:2132:server_put_super()) no obd fs21988-MDT0001 [12003.403976] LustreError: 293947:0:(tgt_mount.c:116:server_deregister_mount()) fs21988-MDT0001 not registered [12003.464238] Lustre: server umount fs21988-MDT0001 complete [12003.467185] LustreError: 293947:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12013.072982] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12013.369274] Lustre: Setting parameter fs21988-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs21988-MDT0001 [12013.434848] Lustre: srv-fs21988-MDT0001: No data found on store. Initialize space. [12013.442964] Lustre: Skipped 4 previous similar messages [12013.625764] Lustre: fs21988-MDT0001: new disk, initializing [12013.629153] Lustre: Skipped 3 previous similar messages [12013.747777] Lustre: srv-fs21988-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12013.750203] Lustre: 294413:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21988-MDT0001-mdtlov: can't create new llog: rc = -115 [12013.755168] LustreError: 294413:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21988-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12019.276240] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12026.546949] Lustre: server umount fs21988-MDT0001 complete [12034.548506] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12046.048489] loop1: detected capacity change from 0 to 204800000 [12049.902387] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12052.309736] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12052.641415] Lustre: 295982:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21988-MDT0001-mdtlov: can't create new llog: rc = -115 [12052.652154] LustreError: 295982:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21988-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12057.510671] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12063.734145] Lustre: server umount fs21988-MDT0001 complete [12075.495759] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12075.506585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12075.518916] Lustre: Skipped 4 previous similar messages [12075.541076] Lustre: Skipped 10 previous similar messages [12080.443926] Lustre: server umount lustre-MDT0000 complete [12089.884811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12090.064317] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12095.148676] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12095.472393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12104.001501] loop1: detected capacity change from 0 to 204800000 [12110.753414] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12111.501738] Lustre: 298430:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21988-MDT0001-mdtlov: can't create new llog: rc = -115 [12111.518501] LustreError: 298430:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21988-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12116.949876] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12124.113914] Lustre: server umount fs21988-MDT0001 complete [12156.924842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12156.943685] Lustre: Skipped 2 previous similar messages [12170.208472] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12170.327189] Lustre: server umount lustre-OST0000 complete [12177.388316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12189.152170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12189.597583] Lustre: server umount lustre-MDT0000 complete [12193.582288] LustreError: 290969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625416 with bad export cookie 10758078393336485176 [12193.591120] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12193.597143] LustreError: 290969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12194.211121] Lustre: server umount lustre-MDT0001 complete [12207.909987] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12211.086662] Key type lgssc unregistered [12211.391534] LNet: 300780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12211.407815] LNetError: 300780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12211.433442] LNet: Removed LNI 192.168.202.137@tcp [12212.459276] Key type .llcrypt unregistered [12212.460531] Key type ._llcrypt unregistered [12234.775807] Key type ._llcrypt registered [12234.781755] Key type .llcrypt registered [12234.898976] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [12235.826802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12235.889693] alg: No test for adler32 (adler32-zlib) [12237.045740] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [12237.296880] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [12239.024280] Key type lgssc registered [12240.356675] Lustre: Echo OBD driver; http://www.lustre.org/ [12249.306661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12249.342336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12250.960680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12256.549016] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12261.370013] LustreError: 302052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12261.393145] LustreError: 302052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12266.472477] LustreError: 302053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12266.889366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12267.459794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12272.543973] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12281.865172] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12288.955735] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12296.635591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12297.190513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12305.850476] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12312.561253] LustreError: 303883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12313.986345] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12317.672130] LustreError: 303883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12321.785201] LustreError: 303881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12321.811985] LustreError: 303881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12330.529920] LustreError: 304555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12330.551950] LustreError: 304555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12345.126853] loop1: detected capacity change from 0 to 204800000 [12345.426341] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12345.444838] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12345.464839] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12347.375628] LustreError: 303881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12347.401438] LustreError: 303881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [12351.180782] loop1: detected capacity change from 0 to 204800000 [12357.691658] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12357.887233] Lustre: fs29612-OST0000: new disk, initializing [12357.893199] Lustre: srv-fs29612-OST0000: No data found on store. Initialize space. [12357.898353] Lustre: 305811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12357.950709] Lustre: fs29612-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12366.142104] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12374.445362] Lustre: server umount fs29612-OST0000 complete [12381.719924] LustreError: 303882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.730178] LustreError: 303882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [12388.709279] loop1: detected capacity change from 0 to 204800000 [12393.469073] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12396.176899] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12396.425779] Lustre: fs29612-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12398.626265] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12404.728828] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12415.826837] loop2: detected capacity change from 0 to 204800000 [12416.149996] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12416.161864] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12416.170750] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12422.325835] loop2: detected capacity change from 0 to 204800000 [12428.861226] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12428.949954] LustreError: fs29612-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12428.959877] LustreError: fs29612-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12428.967774] LustreError: 308853:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [12428.971960] LustreError: 308853:0:(tgt_mount.c:2132:server_put_super()) no obd fs29612-OST0001 [12428.978572] LustreError: 308853:0:(tgt_mount.c:116:server_deregister_mount()) fs29612-OST0001 not registered [12429.008261] Lustre: server umount fs29612-OST0001 complete [12429.011988] LustreError: 308853:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12441.053402] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12441.330842] Lustre: fs29612-OST0001: new disk, initializing [12441.335271] Lustre: srv-fs29612-OST0001: No data found on store. Initialize space. [12441.346814] Lustre: 309405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12441.466311] Lustre: fs29612-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12447.886890] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12448.287240] LustreError: 303883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12448.315127] LustreError: 303883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12457.729838] Lustre: server umount fs29612-OST0000 complete [12470.022364] Lustre: server umount fs29612-OST0001 complete [12484.272638] loop1: detected capacity change from 0 to 204800000 [12491.155286] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12491.472028] Lustre: fs29612-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12498.502919] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12506.455576] loop2: detected capacity change from 0 to 204800000 [12512.372509] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12519.655906] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12528.841478] Lustre: server umount fs29612-OST0000 complete [12538.478235] Lustre: server umount fs29612-OST0001 complete [12547.058900] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12552.204364] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12557.294091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12557.294183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12557.347795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12557.354591] Lustre: Skipped 1 previous similar message [12560.953490] Lustre: server umount lustre-OST0000 complete [12567.523115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12567.524039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12567.530655] Lustre: Skipped 2 previous similar messages [12567.543541] Lustre: Skipped 1 previous similar message [12571.526292] Lustre: server umount lustre-MDT0000 complete [12575.649510] LustreError: 302032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625798 with bad export cookie 15196411740480897808 [12575.658154] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12575.669979] LustreError: 302032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12576.012548] Lustre: server umount lustre-MDT0001 complete [12589.711567] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12593.570126] Key type lgssc unregistered [12593.919856] LNet: 315329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12593.927870] LNetError: 315329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12593.937793] LNet: Removed LNI 192.168.202.137@tcp [12595.613176] Key type .llcrypt unregistered [12595.615295] Key type ._llcrypt unregistered [12633.222870] Key type ._llcrypt registered [12633.224707] Key type .llcrypt registered [12633.288908] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [12652.651502] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [12654.325734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12654.438204] alg: No test for adler32 (adler32-zlib) [12655.584771] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [12655.853434] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [12657.552889] Key type lgssc registered [12658.780217] Lustre: Echo OBD driver; http://www.lustre.org/ [12694.484914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12694.511623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12695.899938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12695.929213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12695.989579] Lustre: lustre-MDT0000: new disk, initializing [12696.060741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12696.072570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12700.621666] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12716.898931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12717.197278] Lustre: 319194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12717.306395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12717.317683] Lustre: Skipped 1 previous similar message [12717.481391] Lustre: lustre-MDT0001: new disk, initializing [12717.746821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12717.861147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12717.893508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12725.493656] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12735.568513] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12741.662355] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12750.186611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12750.562637] Lustre: lustre-OST0000: new disk, initializing [12750.568774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12750.584293] Lustre: 321165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12750.671380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12757.754941] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12758.146068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12758.165311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12758.246478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12768.196974] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12770.698149] Lustre: Mounted lustre-client - version 2.17.57_80_g74d5152 [12771.800106] Lustre: Unmounted lustre-client [12784.097426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12784.097855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12784.106798] LustreError: Skipped 1 previous similar message [12784.108576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12784.115843] Lustre: Skipped 1 previous similar message [12786.171931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12786.175775] Lustre: Skipped 1 previous similar message [12787.881300] Lustre: server umount lustre-OST0000 complete [12794.339647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12794.346681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12794.364139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12794.376719] Lustre: Skipped 1 previous similar message [12796.391832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12796.394634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12798.277862] Lustre: server umount lustre-MDT0000 complete [12801.908065] LustreError: 319186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626025 with bad export cookie 8848385027242556403 [12801.912992] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12802.428221] Lustre: server umount lustre-MDT0001 complete [12815.437975] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12818.423814] Key type lgssc unregistered [12824.994398] LNet: 323329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12825.003455] LNetError: 323329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12899.757427] LNet: Removed LNI 192.168.202.137@tcp [12900.735191] Key type .llcrypt unregistered [12900.738947] Key type ._llcrypt unregistered [12919.833103] Key type ._llcrypt registered [12919.834109] Key type .llcrypt registered [12919.936949] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [12935.952807] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [12937.263504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12937.291785] alg: No test for adler32 (adler32-zlib) [12938.350913] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [12938.522660] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [12940.176163] Key type lgssc registered [12941.817465] Lustre: Echo OBD driver; http://www.lustre.org/ [12978.921539] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:50:00 (1787626200) [12986.259456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12986.299299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12987.717648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12987.740790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12987.850364] Lustre: lustre-MDT0000: new disk, initializing [12987.983869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12988.026653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12992.311444] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13001.872030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13002.001760] Lustre: 326554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13002.061445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13002.063946] Lustre: Skipped 1 previous similar message [13002.105418] Lustre: lustre-MDT0001: new disk, initializing [13002.156448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13002.186712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13002.195826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13007.072164] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13019.378422] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13029.509307] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13037.734960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13037.929655] Lustre: lustre-OST0000: new disk, initializing [13037.936736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13037.948159] Lustre: 328525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13038.022596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13043.928840] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13045.302107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13045.311235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13045.379321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13054.004393] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13075.961636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13075.971596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13080.686058] Lustre: server umount lustre-OST0000 complete [13086.180732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13086.185726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13086.207029] Lustre: Skipped 2 previous similar messages [13086.213389] Lustre: Skipped 1 previous similar message [13091.301447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13091.307872] Lustre: Skipped 1 previous similar message [13096.418057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13096.432573] Lustre: Skipped 1 previous similar message [13100.000180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13100.160192] Lustre: server umount lustre-MDT0000 complete [13103.096037] LustreError: 326545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626326 with bad export cookie 2693010073475623838 [13103.102076] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13103.107320] LustreError: 326545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13103.351202] Lustre: server umount lustre-MDT0001 complete [13113.144458] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13115.596868] Key type lgssc unregistered [13115.971439] LNet: 330438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13115.988023] LNetError: 330438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13116.005225] LNet: Removed LNI 192.168.202.137@tcp [13116.940900] Key type .llcrypt unregistered [13116.943292] Key type ._llcrypt unregistered [13142.772690] Key type ._llcrypt registered [13142.774767] Key type .llcrypt registered [13142.874309] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13143.899369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13143.907485] alg: No test for adler32 (adler32-zlib) [13144.964870] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [13145.208841] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13146.920144] Key type lgssc registered [13148.154134] Lustre: Echo OBD driver; http://www.lustre.org/ [13155.469675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13155.482710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13157.010478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13160.703508] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13167.587036] LustreError: 331712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13167.597804] LustreError: 331712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13170.081588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13170.330534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13175.851669] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13184.294510] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13190.728946] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13197.072935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13197.445922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13203.889887] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13211.692781] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13224.754627] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13226.500829] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13232.097926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13232.105394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13232.117710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13234.234445] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [13235.687786] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13235.689574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13235.709224] Lustre: Skipped 2 previous similar messages [13237.028992] Lustre: server umount lustre-MDT0000 complete [13239.329585] LustreError: 331706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13240.811694] LustreError: 331706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13240.852890] LustreError: 331706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13241.419696] LustreError: 331692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626464 with bad export cookie 17300808600141381655 [13241.431077] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13241.436226] LustreError: 331692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13244.449406] LustreError: 331707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13244.466194] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [13245.928783] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13245.944413] Lustre: Skipped 1 previous similar message [13247.914126] Lustre: server umount lustre-MDT0001 complete [13253.123940] Lustre: server umount lustre-OST0000 complete [13264.501740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13269.118434] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13271.935652] Lustre: 335895:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.137@tcp [13274.292643] Lustre: server umount MGS complete [13283.909613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13284.383458] LustreError: 336462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13284.412553] LustreError: 336462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13284.623753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13289.259275] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13294.052230] LustreError: 336468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13294.074345] LustreError: 336468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13298.364571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13298.642863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13303.315818] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13311.966941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13312.242212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13320.392856] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13329.364108] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13331.054700] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13336.043619] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13337.452222] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13343.721611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13343.729133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13343.744622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13343.748254] Lustre: Skipped 2 previous similar messages [13344.651311] Lustre: server umount lustre-OST0000 complete [13349.862623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13349.877066] LustreError: Skipped 1 previous similar message [13349.887603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13349.904987] Lustre: Skipped 1 previous similar message [13355.491811] LustreError: 338159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13355.510873] LustreError: 338159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13355.530520] Lustre: server umount lustre-MDT0000 complete [13359.207770] LustreError: 336448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626582 with bad export cookie 17300808600141382495 [13359.215768] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13359.220501] LustreError: 336448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13359.585592] Lustre: server umount lustre-MDT0001 complete [13372.341482] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13375.400678] Key type lgssc unregistered [13375.709731] LNet: 339909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13375.731928] LNetError: 339909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13375.760763] LNet: Removed LNI 192.168.202.137@tcp [13376.785941] Key type .llcrypt unregistered [13376.792770] Key type ._llcrypt unregistered [13400.635417] Key type ._llcrypt registered [13400.640409] Key type .llcrypt registered [13400.792187] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [13419.354048] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13420.543448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13420.702062] alg: No test for adler32 (adler32-zlib) [13421.726639] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [13421.956257] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13423.632243] Key type lgssc registered [13424.921786] Lustre: Echo OBD driver; http://www.lustre.org/ [13460.487466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13460.528850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13461.821441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13461.875499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13461.971253] Lustre: lustre-MDT0000: new disk, initializing [13462.053518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13462.074689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13467.207507] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13479.928417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13480.047824] Lustre: 343072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13480.103181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13480.107487] Lustre: Skipped 1 previous similar message [13480.169464] Lustre: lustre-MDT0001: new disk, initializing [13480.294299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13480.331431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13480.350917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13484.916995] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13495.177387] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13501.908441] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13509.289223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13509.642483] Lustre: lustre-OST0000: new disk, initializing [13509.647625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13509.657931] Lustre: 345038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13509.776290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13516.763781] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13519.438424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13519.451226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13519.606640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13527.077985] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13531.688478] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13531.940701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13536.418428] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13536.718426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13540.321396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13540.331871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13540.346740] LustreError: Skipped 1 previous similar message [13540.349423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13540.360117] Lustre: Skipped 1 previous similar message [13540.379624] Lustre: Skipped 1 previous similar message [13545.463624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13545.471069] Lustre: Skipped 1 previous similar message [13550.566782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13550.574821] Lustre: Skipped 1 previous similar message [13553.123347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13553.286282] Lustre: server umount lustre-OST0000 complete [13557.228296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13557.237875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13557.254536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13560.807882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13563.523418] Lustre: server umount lustre-MDT0000 complete [13568.045409] LustreError: 343064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626791 with bad export cookie 17250714226512883003 [13568.058864] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13568.062887] LustreError: 343064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13568.514856] Lustre: server umount lustre-MDT0001 complete [13579.212369] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 23:00:00 (1787626800) [13597.788844] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [13610.870623] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13658.836697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13659.115569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13659.146583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13659.225357] Lustre: lustre-MDT0000: new disk, initializing [13659.292303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13659.312415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13664.462302] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13675.426985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13675.566982] Lustre: 349815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13675.621561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13675.625719] Lustre: Skipped 1 previous similar message [13675.695596] Lustre: lustre-MDT0001: new disk, initializing [13675.794344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13675.824791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13675.835522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13680.629662] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13691.734751] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13698.802156] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13706.638189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13707.046899] Lustre: lustre-OST0000: new disk, initializing [13707.050591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13707.063578] Lustre: 351783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13707.269054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13712.987656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13713.001617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13713.146401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13713.973947] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13724.510737] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13764.588705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13764.598996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13764.653190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13764.665123] Lustre: Skipped 3 previous similar messages [13774.822514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13774.825248] Lustre: Skipped 3 previous similar messages [13777.888143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13778.364951] Lustre: server umount lustre-OST0000 complete [13783.532894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13783.542448] LustreError: Skipped 1 previous similar message [13783.550913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13783.578178] Lustre: Skipped 1 previous similar message [13789.677392] Lustre: server umount lustre-MDT0000 complete [13790.177866] LustreError: 349824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13790.195405] LustreError: 349824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13794.167984] LustreError: 349807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787627017 with bad export cookie 17250714226512883885 [13794.180604] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13794.186989] LustreError: 349807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13794.845125] Lustre: server umount lustre-MDT0001 complete [13807.537323] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13810.743730] Key type lgssc unregistered [13810.999821] LNet: 353962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13811.011643] LNetError: 353962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13811.035182] LNet: Removed LNI 192.168.202.137@tcp [13812.059364] Key type .llcrypt unregistered [13812.065777] Key type ._llcrypt unregistered [13913.447945] Key type ._llcrypt registered [13913.449913] Key type .llcrypt registered [13913.529958] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13913.853436] Key type .llcrypt unregistered [13913.855171] Key type ._llcrypt unregistered [13996.189769] Key type ._llcrypt registered [13996.203816] Key type .llcrypt registered [13996.367667] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13996.856377] Key type .llcrypt unregistered [13996.862835] Key type ._llcrypt unregistered [14028.546577] Key type ._llcrypt registered [14028.549989] Key type .llcrypt registered [14028.686569] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14029.050697] Key type .llcrypt unregistered [14029.059392] Key type ._llcrypt unregistered [14060.904361] Key type ._llcrypt registered [14060.911439] Key type .llcrypt registered [14061.047662] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14061.415390] Key type .llcrypt unregistered [14061.419952] Key type ._llcrypt unregistered [14147.057614] Key type ._llcrypt registered [14147.063155] Key type .llcrypt registered [14147.296558] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14147.727398] Key type .llcrypt unregistered [14147.729383] Key type ._llcrypt unregistered [14188.897198] Key type ._llcrypt registered [14188.900994] Key type .llcrypt registered [14189.009233] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14189.524627] Key type .llcrypt unregistered [14189.532351] Key type ._llcrypt unregistered [14225.978980] Key type ._llcrypt registered [14225.981879] Key type .llcrypt registered [14226.105339] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14226.520953] Key type .llcrypt unregistered [14226.533804] Key type ._llcrypt unregistered [14261.727356] Key type ._llcrypt registered [14261.732298] Key type .llcrypt registered [14261.921921] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14262.272343] Key type .llcrypt unregistered [14262.274435] Key type ._llcrypt unregistered [14300.278205] Key type ._llcrypt registered [14300.280891] Key type .llcrypt registered [14300.383747] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14300.732466] Key type .llcrypt unregistered [14300.734456] Key type ._llcrypt unregistered [14338.194831] Key type ._llcrypt registered [14338.197265] Key type .llcrypt registered [14338.391267] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14338.996932] Key type .llcrypt unregistered [14339.003534] Key type ._llcrypt unregistered [14421.189778] Key type ._llcrypt registered [14421.191800] Key type .llcrypt registered [14421.330399] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14421.623395] Key type .llcrypt unregistered [14421.625458] Key type ._llcrypt unregistered [14456.870845] Key type ._llcrypt registered [14456.873312] Key type .llcrypt registered [14457.152872] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14457.771436] Key type .llcrypt unregistered [14457.773846] Key type ._llcrypt unregistered [14496.260631] Key type ._llcrypt registered [14496.267511] Key type .llcrypt registered [14496.428558] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14496.853244] Key type .llcrypt unregistered [14496.858457] Key type ._llcrypt unregistered [14567.053856] Key type ._llcrypt registered [14567.055815] Key type .llcrypt registered [14567.255712] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [14587.072526] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [14588.054831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14588.075150] alg: No test for adler32 (adler32-zlib) [14589.575659] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [14589.946759] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [14591.728309] Key type lgssc registered [14592.929040] Lustre: Echo OBD driver; http://www.lustre.org/ [14644.281234] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14644.327871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14645.945246] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14646.038919] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14646.271249] Lustre: tst250fs-MDT0000: new disk, initializing [14646.522278] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14646.583921] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14653.569103] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14667.518537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14667.991900] Lustre: 374376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14668.198905] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14668.214471] Lustre: Skipped 1 previous similar message [14668.481742] Lustre: tst250fs-MDT0001: new disk, initializing [14668.730504] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14668.804664] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14668.829211] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14674.596307] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14680.811604] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14690.027723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14690.576396] Lustre: tst250fs-OST0000: new disk, initializing [14690.585530] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14690.621437] Lustre: 376013:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14690.948074] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14699.644769] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14699.694856] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14699.718932] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14699.906542] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14714.869049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14715.203741] Lustre: tst250fs-OST0001: new disk, initializing [14715.209665] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14715.220201] Lustre: 376886:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14715.479358] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14723.244809] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14723.276725] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14723.487598] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14724.873801] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14731.360575] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14733.561874] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14733.568849] Lustre: Skipped 2 previous similar messages [14754.275186] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14754.277855] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14754.286213] Lustre: Skipped 3 previous similar messages [14754.288252] Lustre: Skipped 2 previous similar messages [14759.221988] LustreError: 378199:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x7366303532747374:0x2:0x0].0x0 (ffff8cbda7be5500) refcount nonzero (1) after lock cleanup; forcing cleanup. [14761.465736] Lustre: server umount tst250fs-MDT0000 complete [14764.545880] LustreError: 375558:0:(ldlm_lib.c:1190:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14764.563961] LustreError: 375558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [14766.030257] LustreError: 374369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787627989 with bad export cookie 18286746673233452240 [14766.032092] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14766.044973] LustreError: 374369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14766.322701] Lustre: server umount tst250fs-MDT0001 complete [14780.964699] Lustre: server umount tst250fs-OST0000 complete [14784.995090] Lustre: 372598:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787627992/real 1787627992] req@ffff8cbc84c9d180 x1874463621716608/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787628008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14785.044304] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14785.646681] Lustre: server umount tst250fs-OST0001 complete [14810.490461] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [14825.213854] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [14865.037371] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 23:21:26 (1787628086) [14866.918823] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14885.897296] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [14907.775791] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [14951.044392] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14671 sec ========= 23:22:50 (1787628170) [14956.161061] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:22:55 (1787628175) === [14962.367198] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:23:02 (1787628182) === [15014.317792] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [15017.800228] Key type lgssc unregistered [15018.176275] LNet: 386049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15018.204052] LNetError: 386049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15018.250408] LNet: Removed LNI 192.168.202.137@tcp [15019.409196] Key type .llcrypt unregistered [15019.413397] Key type ._llcrypt unregistered