[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 909697904 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003214] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008417] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012064] Yama: becoming mindful. [ 0.014041] SELinux: Initializing. [ 0.015076] *** VALIDATE selinux *** [ 0.024645] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031041] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033152] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034108] *** VALIDATE tmpfs *** [ 0.036074] *** VALIDATE proc *** [ 0.037414] *** VALIDATE cgroup *** [ 0.038013] *** VALIDATE cgroup2 *** [ 0.039500] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042030] Spectre V2 : User space: Vulnerable [ 0.043012] Speculative Store Bypass: Vulnerable [ 0.049035] debug: unmapping init [mem 0xffffffffb0659000-0xffffffffb0660fff] [ 0.051300] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.055275] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.056024] ... version: 2 [ 0.057018] ... bit width: 48 [ 0.058016] ... generic registers: 4 [ 0.059024] ... value mask: 0000ffffffffffff [ 0.060016] ... max period: 00007fffffffffff [ 0.061017] ... fixed-purpose events: 3 [ 0.062014] ... event mask: 000000070000000f [ 0.063524] rcu: Hierarchical SRCU implementation. [ 0.066035] smp: Bringing up secondary CPUs ... [ 0.067888] x86: Booting SMP configuration: [ 0.068114] .... node #0, CPUs: #1 #2 #3 [ 0.084153] smp: Brought up 1 node, 4 CPUs [ 0.086265] smpboot: Max logical packages: 1 [ 0.087019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.124039] node 0 deferred pages initialised in 35ms [ 0.134280] devtmpfs: initialized [ 0.135196] x86/mm: Memory block size: 128MB [ 0.154294] gcov: version magic: 0x41383552 [ 0.163325] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.166079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.169234] pinctrl core: initialized pinctrl subsystem [ 0.171193] [ 0.171903] ************************************************************* [ 0.174015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178012] ** ** [ 0.181013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.184014] ** ** [ 0.188013] ** This means that this kernel is built to expose internal ** [ 0.192720] ** IOMMU data structures, which may compromise security on ** [ 0.196012] ** your system. ** [ 0.199011] ** ** [ 0.202013] ** If you see this message and you are not debugging the ** [ 0.205012] ** kernel, report this immediately to your vendor! ** [ 0.207012] ** ** [ 0.209012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.212012] ************************************************************* [ 0.215812] NET: Registered protocol family 16 [ 0.217434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.221060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.224057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.228738] cpuidle: using governor menu [ 0.230862] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.233308] PCI: Using configuration type 1 for base access [ 0.236232] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.243610] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.245018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.249075] cryptd: max_cpu_qlen set to 1000 [ 0.252237] ACPI: Added _OSI(Module Device) [ 0.253013] ACPI: Added _OSI(Processor Device) [ 0.255048] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.258016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.263000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.271079] ACPI: Interpreter enabled [ 0.272082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.273045] ACPI: Using IOAPIC for interrupt routing [ 0.275140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.279421] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.293623] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.296038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.297000] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.303073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.308395] acpiphp: Slot [2] registered [ 0.310111] acpiphp: Slot [5] registered [ 0.312125] acpiphp: Slot [6] registered [ 0.314179] acpiphp: Slot [7] registered [ 0.316103] acpiphp: Slot [8] registered [ 0.317138] acpiphp: Slot [9] registered [ 0.319107] acpiphp: Slot [10] registered [ 0.320112] acpiphp: Slot [3] registered [ 0.322107] acpiphp: Slot [4] registered [ 0.324108] acpiphp: Slot [11] registered [ 0.325093] acpiphp: Slot [12] registered [ 0.327115] acpiphp: Slot [13] registered [ 0.328128] acpiphp: Slot [14] registered [ 0.329102] acpiphp: Slot [15] registered [ 0.330113] acpiphp: Slot [16] registered [ 0.332099] acpiphp: Slot [17] registered [ 0.334092] acpiphp: Slot [18] registered [ 0.335115] acpiphp: Slot [19] registered [ 0.337114] acpiphp: Slot [20] registered [ 0.338127] acpiphp: Slot [21] registered [ 0.340100] acpiphp: Slot [22] registered [ 0.341100] acpiphp: Slot [23] registered [ 0.343137] acpiphp: Slot [24] registered [ 0.344147] acpiphp: Slot [25] registered [ 0.345160] acpiphp: Slot [26] registered [ 0.347123] acpiphp: Slot [27] registered [ 0.349089] acpiphp: Slot [28] registered [ 0.350099] acpiphp: Slot [29] registered [ 0.352104] acpiphp: Slot [30] registered [ 0.353092] acpiphp: Slot [31] registered [ 0.355062] PCI host bridge to bus 0000:00 [ 0.356076] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.358021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.361022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.364025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.366020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.369023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.371182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.375070] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.378000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.378000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.378000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.378000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.380016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.385636] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.388867] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.392035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.395240] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.400012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.411012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.416012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423653] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.459016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.476123] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.485016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.536549] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.545014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.552013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.571015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.582857] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.602035] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.617015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.664016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.693013] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.704016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.711014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.786028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.812575] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.829021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.844309] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.885024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.901933] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.905627] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.908627] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.912514] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.917482] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.923154] iommu: Default domain type: Passthrough [ 0.924596] SCSI subsystem initialized [ 0.927226] ACPI: bus type USB registered [ 0.929232] usbcore: registered new interface driver usbfs [ 0.932120] usbcore: registered new interface driver hub [ 0.935155] usbcore: registered new device driver usb [ 0.938188] pps_core: LinuxPPS API ver. 1 registered [ 0.940013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.944106] PTP clock support registered [ 0.947103] EDAC MC: Ver: 3.0.0 [ 0.949171] PCI: Using ACPI for IRQ routing [ 0.953188] NetLabel: Initializing [ 0.955016] NetLabel: domain hash size = 128 [ 0.957012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.960107] NetLabel: unlabeled traffic allowed by default [ 0.962471] vgaarb: loaded [ 0.965538] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.968019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.974242] clocksource: Switched to clocksource kvm-clock [ 1.125091] VFS: Disk quotas dquot_6.6.0 [ 1.127210] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.132589] *** VALIDATE ramfs *** [ 1.134092] *** VALIDATE hugetlbfs *** [ 1.136092] pnp: PnP ACPI init [ 1.138870] pnp: PnP ACPI: found 6 devices [ 1.199853] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.205271] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.207347] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.209453] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.211780] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.213978] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.217091] NET: Registered protocol family 2 [ 1.219436] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.224411] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.227993] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.235557] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.239564] TCP: Hash tables configured (established 65536 bind 65536) [ 1.242977] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.246671] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.249816] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.253426] NET: Registered protocol family 1 [ 1.257229] RPC: Registered named UNIX socket transport module. [ 1.260054] RPC: Registered udp transport module. [ 1.262195] RPC: Registered tcp transport module. [ 1.264308] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.267245] NET: Registered protocol family 44 [ 1.269062] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.271251] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.273123] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.275610] PCI: CLS 0 bytes, default 64 [ 1.277666] Unpacking initramfs... [ 3.282395] debug: unmapping init [mem 0xffff895f3cc54000-0xffff895f3ffbffff] [ 3.291151] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.301942] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.315174] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.064350] Initialise system trusted keyrings [ 4.066672] Key type blacklist registered [ 4.068647] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.139027] zbud: loaded [ 4.155874] *** VALIDATE nfs *** [ 4.160918] *** VALIDATE nfs4 *** [ 4.167534] pstore: using deflate compression [ 4.182271] Platform Keyring initialized [ 4.310743] NET: Registered protocol family 38 [ 4.313155] Key type asymmetric registered [ 4.314988] Asymmetric key parser 'x509' registered [ 4.317647] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.321486] io scheduler mq-deadline registered [ 4.325147] io scheduler kyber registered [ 4.332808] io scheduler bfq registered [ 4.338996] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.342971] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.346935] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.350716] ACPI: Power Button [PWRF] [ 4.359323] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.366912] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.381871] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.412962] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.472672] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.506712] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.535791] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.541549] Non-volatile memory driver v1.3 [ 4.543687] Linux agpgart interface v0.103 [ 4.577308] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.582429] vda: detected capacity change from 0 to 74702848 [ 4.626420] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.637075] vdb: detected capacity change from 0 to 1073741824 [ 4.661765] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.665655] vdc: detected capacity change from 0 to 2621440000 [ 4.679768] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.682849] vdd: detected capacity change from 0 to 2621440000 [ 4.698195] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.701816] vde: detected capacity change from 0 to 4294967296 [ 4.718217] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.721745] vdf: detected capacity change from 0 to 4294967296 [ 4.728836] libphy: Fixed MDIO Bus: probed [ 4.752635] usbcore: registered new interface driver usbserial_generic [ 4.757264] usbserial: USB Serial support registered for generic [ 4.760679] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.766102] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.767849] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.770912] mousedev: PS/2 mouse device common for all mice [ 4.776454] rtc_cmos 00:05: RTC can wake from S4 [ 4.780141] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.783936] rtc_cmos 00:05: registered as rtc0 [ 4.791283] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.792183] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.799658] intel_pstate: CPU model not supported [ 4.810722] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.821655] hid: raw HID events driver (C) Jiri Kosina [ 4.824846] usbcore: registered new interface driver usbhid [ 4.827747] usbhid: USB HID core driver [ 4.830071] drop_monitor: Initializing network drop monitor service [ 4.833661] Initializing XFRM netlink socket [ 4.835993] NET: Registered protocol family 10 [ 4.839928] Segment Routing with IPv6 [ 4.842148] NET: Registered protocol family 17 [ 4.844977] mpls_gso: MPLS GSO support [ 4.852172] RAS: Correctable Errors collector initialized. [ 4.855104] AVX version of gcm_enc/dec engaged. [ 4.857391] AES CTR mode by8 optimization enabled [ 4.973299] sched_clock: Marking stable (4973275471, 0)->(6800205232, -1826929761) [ 5.009430] registered taskstats version 1 [ 5.012697] Loading compiled-in X.509 certificates [ 5.015540] zswap: loaded using pool lzo/zbud [ 5.056686] Key type big_key registered [ 5.077178] Key type encrypted registered [ 5.079204] ima: No TPM chip found, activating TPM-bypass! [ 5.082244] ima: Allocated hash algorithm: sha1 [ 5.084968] ima: No architecture policies found [ 5.091283] evm: Initialising EVM extended attributes: [ 5.093938] evm: security.selinux [ 5.096078] evm: security.ima [ 5.097603] evm: security.capability [ 5.099240] evm: HMAC attrs: 0x1 [ 5.102894] rtc_cmos 00:05: setting system clock to 2026-08-10 01:45:03 UTC (1786326303) [ 5.110758] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 5.115332] debug: unmapping init [mem 0xffffffffb0382000-0xffffffffb0658fff] [ 5.124534] Write protecting the kernel read-only data: 28672k [ 5.129526] debug: unmapping init [mem 0xffffffffaea03000-0xffffffffaebfffff] [ 5.135096] debug: unmapping init [mem 0xffffffffaf314000-0xffffffffaf3fffff] [ 5.233712] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.259650] systemd[1]: Detected virtualization kvm. [ 5.264453] systemd[1]: Detected architecture x86-64. [ 5.266102] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.314750] systemd[1]: No hostname configured. [ 5.318680] systemd[1]: Set hostname to . [ 5.323185] random: systemd: uninitialized urandom read (16 bytes read) [ 5.328868] systemd[1]: Initializing machine ID from random generator. [ 5.683628] random: ln: uninitialized urandom read (6 bytes read) [ 5.943330] random: systemd: uninitialized urandom read (16 bytes read) [ 5.980765] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.988577] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.999821] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.152345] device-mapper: uevent: version 1.0.3 [ 8.158680] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.066166] virtio_net virtio0 ens2: renamed from eth0 [ 10.504555] random: fast init done [ 11.332127] scsi host0: ata_piix [ 11.437414] scsi host1: ata_piix [ 11.441718] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.445659] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.760655] random: crng init done [ 16.765919] random: 7 urandom warning(s) missed due to ratelimiting [ 19.662532] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.064246] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.130530] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.746702] SELinux: Disabled at runtime. [ 24.836964] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.845613] systemd[1]: Detected virtualization kvm. [ 24.847839] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.625412] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.630460] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.643393] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.648582] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.655907] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.666769] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.677724] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ 27.483647] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 28.664119] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.819509] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.856561] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.634667] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.783809] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 37.452086] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit)[ 38.377425] NFS: Registering the id_resolver key type [ 38.380733] Key type id_resolver registered [ 38.382872] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg328-server login: [ 84.634983] hrtimer: interrupt took 2243464 ns [ 96.278194] libcfs: loading out-of-tree module taints kernel. [ 96.312585] Key type ._llcrypt registered [ 96.316232] Key type .llcrypt registered [ 96.401678] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 113.052820] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 114.636619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.648572] alg: No test for adler32 (adler32-zlib) [ 116.326896] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 117.224208] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 119.051830] Key type lgssc registered [ 120.558233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.693279] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 173.899126] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 185.428836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.445856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.844362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.881927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.966624] Lustre: lustre-MDT0000: new disk, initializing [ 187.038233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.057789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.610941] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.680654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.888349] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.959959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.970556] Lustre: Skipped 1 previous similar message [ 204.074428] Lustre: lustre-MDT0001: new disk, initializing [ 204.180319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.273183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.297171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.012589] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.384975] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.574273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.782787] Lustre: lustre-OST0000: new disk, initializing [ 220.786191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.790517] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.852173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.600424] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.036771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.050686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 229.112868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.335448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.426613] Lustre: lustre-OST0001: new disk, initializing [ 241.431235] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 241.437852] Lustre: 9522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.505509] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 247.892930] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.898039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 249.907548] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 249.977755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 260.920601] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 271.629635] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 278.091782] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.024639] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing yml_node [ 287.679386] Lustre: DEBUG MARKER: Client: 2.17.56.55 [ 289.647564] Lustre: DEBUG MARKER: MDS: 2.17.56.55 [ 292.317604] Lustre: DEBUG MARKER: OSS: 2.17.56.55 [ 293.984856] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 9 21:49:52 EDT 2026 [ 311.577986] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 313.278410] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 331.744297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.748016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.764931] Lustre: Skipped 3 previous similar messages [ 331.814628] Lustre: Skipped 3 previous similar messages [ 333.875911] Lustre: server umount lustre-MDT0000 complete [ 337.478692] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786326635 with bad export cookie 11411100322284570067 [ 337.483265] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 337.489902] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 341.985394] LustreError: 7796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.001288] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.022890] LustreError: 7796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 342.039613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.008103] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.020779] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 343.980633] Lustre: server umount lustre-MDT0001 complete [ 346.840646] Lustre: server umount lustre-OST0000 complete [ 350.348866] Lustre: server umount lustre-OST0001 complete [ 357.329127] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 365.372924] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 403.162876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.328685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 403.446783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 403.523868] Lustre: lustre-MDT0000: new disk, initializing [ 403.636604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.655733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 409.376440] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 421.061818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.138209] Lustre: 15076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.179662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.185144] Lustre: Skipped 1 previous similar message [ 421.326800] Lustre: lustre-MDT0001: new disk, initializing [ 421.427715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.454628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.473684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 426.127638] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 436.812596] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 442.516412] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 451.525780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 451.845258] Lustre: lustre-OST0000: new disk, initializing [ 451.854067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 451.861193] Lustre: 17044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 452.253460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.464460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 453.494041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 458.737260] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 458.755177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 467.276641] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 472.389640] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 472.556366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 477.250494] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 477.424706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 484.368395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.373618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 484.429017] Lustre: Skipped 1 previous similar message [ 484.429342] LustreError: Skipped 1 previous similar message [ 484.445641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.451893] Lustre: Skipped 4 previous similar messages [ 485.624886] Lustre: server umount lustre-OST0000 complete [ 489.447156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.450389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 489.454466] Lustre: Skipped 1 previous similar message [ 489.461817] Lustre: Skipped 1 previous similar message [ 494.582260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.639722] Lustre: server umount lustre-MDT0000 complete [ 499.059760] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786326797 with bad export cookie 11411100322284571593 [ 499.062235] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.068070] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 499.381585] Lustre: server umount lustre-MDT0001 complete [ 504.504557] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:53:22 (1786326802) [ 526.858165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.396525] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.475760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.565316] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 532.973576] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.066201] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.275159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.395648] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 555.596993] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 561.855960] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 564.628735] Lustre: Failing over lustre-MDT0000 [ 564.710112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.720360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 564.723732] Lustre: Skipped 1 previous similar message [ 565.205695] Lustre: server umount lustre-MDT0000 complete [ 566.817483] LustreError: 20916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.825104] LustreError: 20916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 583.135126] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786326865/real 1786326865] req@ffff895fb78c2300 x1873099007092096/t0(0) o400->MGC192.168.203.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786326881 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 583.158948] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 583.184020] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.208978] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 585.038076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.399996] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff895f848ab100 x1873099007095040/t0(0) o250->MGC192.168.203.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 594.731967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.739729] Lustre: Skipped 1 previous similar message [ 600.068656] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 600.178124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 600.198108] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 600.259918] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 611.644328] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 616.121334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.139383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 616.141377] Lustre: Skipped 2 previous similar messages [ 616.156965] Lustre: Skipped 1 previous similar message [ 622.407865] Lustre: server umount lustre-MDT0000 complete [ 626.379540] LustreError: 20652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786326924 with bad export cookie 11411100322284573014 [ 626.388644] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 626.396701] LustreError: 20652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 626.885941] Lustre: server umount lustre-MDT0001 complete [ 637.207408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.589592] LustreError: 23258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.602368] LustreError: 23258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 641.866567] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 651.085139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.258507] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 663.417358] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 669.947268] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 672.742763] Lustre: Failing over lustre-MDT0000 [ 672.911868] Lustre: server umount lustre-MDT0000 complete [ 673.253448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.274979] Lustre: Skipped 1 previous similar message [ 688.799200] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786326971/real 1786326971] req@ffff895e83a04a80 x1873099007122176/t0(0) o400->MGC192.168.203.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786326987 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 688.828264] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 693.148761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.359987] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff895f9e378000 x1873099007125120/t0(0) o250->MGC192.168.203.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 699.371363] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 699.439877] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 699.454921] Lustre: MGS: Client 7b1fe78c-bc36-49a1-acfd-a33db1ceb51b (at 0@lo) reconnecting [ 699.466320] Lustre: MGC192.168.203.128@tcp: Connection restored to 0@lo (at 0@lo) [ 699.474898] Lustre: Skipped 1 previous similar message [ 699.802648] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 704.314264] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 704.998326] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 705.025257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 705.032934] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 712.790440] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 720.379369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 720.391135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.444314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 720.452048] Lustre: Skipped 3 previous similar messages [ 723.260613] Lustre: server umount lustre-MDT0000 complete [ 727.013580] LustreError: 23258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 727.033802] LustreError: 23258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 727.080231] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786327025 with bad export cookie 11411100322284573952 [ 727.092621] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 727.092939] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 727.408146] Lustre: server umount lustre-MDT0001 complete [ 738.821819] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:57:16 (1786327036) [ 756.351146] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 766.208665] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 808.506663] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 822.920612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.541916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 823.674666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 823.813566] Lustre: lustre-MDT0000: new disk, initializing [ 823.955148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.964320] Lustre: Skipped 3 previous similar messages [ 823.993633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 829.507428] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 841.583203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.830465] Lustre: 29628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 842.276333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 847.591136] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 857.782985] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 863.915748] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 872.192602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.483199] Lustre: 31600:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 874.218407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 874.241952] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 874.281865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 874.291904] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 879.915178] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 889.943631] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 895.508921] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 915.427607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 915.434431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.439260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 915.444464] Lustre: Skipped 2 previous similar messages [ 915.449751] Lustre: Skipped 2 previous similar messages [ 920.582538] Lustre: server umount lustre-OST0000 complete [ 924.129193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 930.070556] Lustre: server umount lustre-MDT0000 complete [ 930.784558] LustreError: 30504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.798363] LustreError: 30504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 935.166651] LustreError: 29622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786327233 with bad export cookie 11411100322284574365 [ 935.176034] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 935.183735] LustreError: 29622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 935.861618] Lustre: server umount lustre-MDT0001 complete [ 947.353561] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 950.894777] Key type lgssc unregistered [ 951.145673] LNet: 33582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.157541] LNetError: 33582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.205090] LNet: Removed LNI 192.168.203.128@tcp [ 952.164166] Key type .llcrypt unregistered [ 952.167318] Key type ._llcrypt unregistered [ 976.449262] Key type ._llcrypt registered [ 976.451463] Key type .llcrypt registered [ 976.529812] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 977.563362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 977.669083] alg: No test for adler32 (adler32-zlib) [ 978.976444] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 979.146336] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 980.807210] Key type lgssc registered [ 982.161669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 989.572561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 989.611414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.971333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.776698] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1001.442883] LustreError: 35007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.466347] LustreError: 35007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1002.466419] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.466503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.754608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1010.875177] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1019.190937] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1025.997510] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1033.387926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1033.650157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.850983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1040.185851] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1048.438450] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1051.192280] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1067.176585] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:02:44 (1786327364) [ 1069.722474] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1071.837787] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1079.627954] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:02:57 (1786327377) [ 1087.146091] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:03:05 (1786327385) [ 1088.112326] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1096.476423] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1096.482148] Lustre: Skipped 2 previous similar messages [ 1098.040546] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:03:16 (1786327396) [ 1100.714701] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1100.720613] Lustre: Skipped 2 previous similar messages [ 1109.430189] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1109.437427] Lustre: Skipped 4 previous similar messages [ 1124.090477] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:03:42 (1786327422) [ 1126.120410] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1129.471568] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1137.314653] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:03:55 (1786327435) [ 1138.990344] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1138.993502] Lustre: Skipped 6 previous similar messages [ 1151.717554] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:04:09 (1786327449) [ 1164.768353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1164.774653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.782392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.865284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.880225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1168.882167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.513711] Lustre: server umount lustre-OST0000 complete [ 1175.009510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1175.015529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.022988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1175.027166] Lustre: Skipped 1 previous similar message [ 1179.110801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.111668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.926898] Lustre: server umount lustre-MDT0000 complete [ 1184.228939] LustreError: 35736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.298639] LustreError: 35736:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1185.088663] LustreError: 34988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786327483 with bad export cookie 15236033282572541346 [ 1185.091841] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1185.119658] LustreError: 34988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1185.336987] Lustre: server umount lustre-MDT0001 complete [ 1197.276988] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1199.985462] Key type lgssc unregistered [ 1200.180700] LNet: 42856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.185332] LNetError: 42856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.253935] LNet: Removed LNI 192.168.203.128@tcp [ 1202.016580] Key type .llcrypt unregistered [ 1202.018472] Key type ._llcrypt unregistered [ 1219.237550] Key type ._llcrypt registered [ 1219.239658] Key type .llcrypt registered [ 1219.418830] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1233.844796] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1235.913553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1235.943995] alg: No test for adler32 (adler32-zlib) [ 1237.450526] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1237.613695] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1239.231389] Key type lgssc registered [ 1240.680169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1275.891961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1275.962748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.452380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1277.521381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1277.574679] Lustre: lustre-MDT0000: new disk, initializing [ 1277.711190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.741810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1283.025575] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1296.299099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.463825] Lustre: 46012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1296.616995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1296.631396] Lustre: Skipped 1 previous similar message [ 1296.827559] Lustre: lustre-MDT0001: new disk, initializing [ 1296.956935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1297.093553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1297.104340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1302.821514] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1312.172395] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1317.490409] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1325.227689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1325.651677] Lustre: lustre-OST0000: new disk, initializing [ 1325.660363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1325.666688] Lustre: 47987:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1325.769590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.932934] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1332.842779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1332.853911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1333.024069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1342.495148] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1347.398797] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1347.598443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1351.612269] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1351.772273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1353.696187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1353.699968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.710893] LustreError: Skipped 1 previous similar message [ 1353.723277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1353.729070] Lustre: Skipped 1 previous similar message [ 1358.817873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1358.825730] Lustre: Skipped 1 previous similar message [ 1359.718442] Lustre: server umount lustre-OST0000 complete [ 1363.938883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1363.943299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.025163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1364.037749] Lustre: Skipped 2 previous similar messages [ 1364.058092] Lustre: Skipped 1 previous similar message [ 1369.057711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1369.063717] Lustre: Skipped 1 previous similar message [ 1369.222029] Lustre: server umount lustre-MDT0000 complete [ 1373.635228] LustreError: 46005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786327672 with bad export cookie 7049028506924495445 [ 1373.638049] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1373.645654] LustreError: 46005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1373.955770] Lustre: server umount lustre-MDT0001 complete [ 1385.941607] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:08:03 (1786327683) [ 1393.520614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.030699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.308079] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.424521] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1399.651902] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1404.391107] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.423104] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1408.418929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.640228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1413.967450] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1422.655698] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1428.926360] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1436.433199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1436.649294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.335820] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1450.093158] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1458.249231] LustreError: 52976:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1459.384737] Lustre: Modifying parameter general.timeout=1 in log params [ 1460.470974] Lustre: Modifying parameter general.timeout=2 in log params [ 1462.698416] Lustre: Modifying parameter general.timeout=5 in log params [ 1462.705370] Lustre: Skipped 2 previous similar messages [ 1468.191502] Lustre: Modifying parameter general.timeout=11 in log params [ 1468.194692] Lustre: Skipped 5 previous similar messages [ 1476.524547] Lustre: Modifying parameter general.timeout=21 in log params [ 1476.534302] Lustre: Skipped 9 previous similar messages [ 1493.564804] Lustre: Modifying parameter general.timeout=40 in log params [ 1493.568582] Lustre: Skipped 18 previous similar messages [ 1526.295644] Lustre: Modifying parameter general.timeout=74 in log params [ 1526.310763] Lustre: Skipped 33 previous similar messages [ 1590.955516] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:11:28 (1786327888) [ 1591.948789] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1591.954679] Lustre: Skipped 56 previous similar messages [ 1593.728162] Lustre: 59919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1595.944479] Lustre: 60030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1597.960489] Lustre: 60131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1603.347992] Lustre: 60425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1603.356665] Lustre: 60425:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1630.004799] Lustre: 61784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1630.031354] LustreError: 61784:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1634.652900] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:12:12 (1786327932) [ 1648.097673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.107150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.109607] Lustre: Skipped 2 previous similar messages [ 1648.124343] Lustre: Skipped 2 previous similar messages [ 1653.020518] Lustre: server umount lustre-MDT0000 complete [ 1653.224918] LustreError: 53914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1653.246614] LustreError: 53914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1656.587513] LustreError: 52290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786327954 with bad export cookie 7049028506924496327 [ 1656.591422] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1656.595383] LustreError: 52290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1656.838599] Lustre: server umount lustre-MDT0001 complete [ 1671.913450] Lustre: server umount lustre-OST0000 complete [ 1699.729655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.910512] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1700.088948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.884434] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1712.616703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.684867] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1712.707445] Lustre: 64319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1717.108692] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1724.671280] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1731.753232] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1740.328101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1740.437692] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1740.709042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.716535] Lustre: Skipped 1 previous similar message [ 1746.061489] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1754.541309] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1757.539139] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1757.571379] Lustre: Skipped 22 previous similar messages [ 1757.697371] Lustre: 66884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1764.837828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1764.858244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.877884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1769.445570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1769.446697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.110136] Lustre: server umount lustre-OST0000 complete [ 1779.691355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1779.692327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.704064] Lustre: Skipped 1 previous similar message [ 1781.947939] Lustre: server umount lustre-MDT0000 complete [ 1784.801826] LustreError: 66186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1784.839972] LustreError: 66186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1785.813186] LustreError: 64310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786328084 with bad export cookie 7049028506924499316 [ 1785.818290] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.835823] LustreError: 64310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1786.086544] Lustre: server umount lustre-MDT0001 complete [ 1798.022702] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1800.837712] Key type lgssc unregistered [ 1801.074803] LNet: 68096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.099641] LNetError: 68096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.150358] LNet: Removed LNI 192.168.203.128@tcp [ 1803.153202] Key type .llcrypt unregistered [ 1803.155235] Key type ._llcrypt unregistered [ 1830.432241] Key type ._llcrypt registered [ 1830.434378] Key type .llcrypt registered [ 1830.532569] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1831.760573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1831.769786] alg: No test for adler32 (adler32-zlib) [ 1833.291280] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1833.773045] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1835.489246] Key type lgssc registered [ 1836.421678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1844.998108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1845.013504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.385782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.792265] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1856.037760] LustreError: 69512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1856.083180] LustreError: 69512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1860.051298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.495984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1864.502602] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1873.696443] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1879.691522] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1887.378095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1888.072809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.654433] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1902.142185] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1913.587952] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1922.018529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1922.029437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.047292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.119884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.120952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.136501] Lustre: Skipped 1 previous similar message [ 1926.145299] Lustre: Skipped 1 previous similar message [ 1928.148438] Lustre: server umount lustre-MDT0000 complete [ 1931.243922] LustreError: 71816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.265374] LustreError: 71816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1931.813534] LustreError: 69493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786328230 with bad export cookie 14224177416584545718 [ 1931.820805] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1931.822465] LustreError: 69493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1932.180192] Lustre: server umount lustre-MDT0001 complete [ 1946.244826] Lustre: server umount lustre-OST0000 complete [ 1973.700918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.904860] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1973.927331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1974.114430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.131286] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1985.947898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.025267] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1986.043529] Lustre: 74382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1986.426239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1990.535838] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1997.071649] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2002.863978] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2011.561994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2011.671635] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2012.011052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.797945] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2025.843147] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2028.294470] Lustre: 76956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2030.184251] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2034.657076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2034.676662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.705304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2034.718927] Lustre: Skipped 1 previous similar message [ 2040.308987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2040.313355] Lustre: Skipped 2 previous similar messages [ 2040.630423] Lustre: server umount lustre-OST0000 complete [ 2045.420878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.423582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.442198] Lustre: Skipped 2 previous similar messages [ 2045.462019] Lustre: Skipped 2 previous similar messages [ 2050.516265] Lustre: server umount lustre-MDT0000 complete [ 2050.542476] LustreError: 74393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.570024] LustreError: 74393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2053.941386] LustreError: 76249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786328352 with bad export cookie 14224177416584546551 [ 2053.951650] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2054.244098] Lustre: server umount lustre-MDT0001 complete [ 2064.568539] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2067.419856] Key type lgssc unregistered [ 2067.708292] LNet: 78171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.718713] LNetError: 78171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.773456] LNet: Removed LNI 192.168.203.128@tcp [ 2070.024249] Key type .llcrypt unregistered [ 2070.027592] Key type ._llcrypt unregistered [ 2094.672905] Key type ._llcrypt registered [ 2094.675348] Key type .llcrypt registered [ 2094.754095] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2096.692133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.770108] alg: No test for adler32 (adler32-zlib) [ 2098.098649] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2098.266888] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2099.959159] Key type lgssc registered [ 2101.198905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2107.995863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2108.014892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.464989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.665949] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2118.628225] LustreError: 79584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.644403] LustreError: 79584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2122.955625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.481936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2128.674568] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2135.828405] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2141.756067] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2149.092581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2149.546834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.912727] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2163.991575] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2173.861149] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2175.067240] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2176.614807] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2176.628137] Lustre: Skipped 1 previous similar message [ 2178.763729] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2178.770561] Lustre: Skipped 1 previous similar message [ 2183.399703] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2183.404135] Lustre: Skipped 4 previous similar messages [ 2191.970668] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2191.974573] Lustre: Skipped 8 previous similar messages [ 2208.104905] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2208.109721] Lustre: Skipped 16 previous similar messages [ 2240.793815] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2240.798339] Lustre: Skipped 32 previous similar messages [ 2278.403394] Lustre: 87687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2281.362744] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:22:59 (1786328579) [ 2283.789171] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2285.688498] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:23:03 (1786328583) [ 2291.172202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.181044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2291.185215] Lustre: Skipped 1 previous similar message [ 2296.146340] Lustre: server umount lustre-OST0000 complete [ 2301.414877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.420540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2301.421672] Lustre: Skipped 2 previous similar messages [ 2301.430658] Lustre: Skipped 1 previous similar message [ 2305.957776] Lustre: server umount lustre-MDT0000 complete [ 2306.530554] LustreError: 79583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.574195] LustreError: 79583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2310.289356] LustreError: 80323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786328608 with bad export cookie 6294212015495550818 [ 2310.297726] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2310.551565] Lustre: server umount lustre-MDT0001 complete [ 2322.740922] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2325.267648] Key type lgssc unregistered [ 2325.515863] LNet: 88937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.522749] LNetError: 88937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.532376] LNet: Removed LNI 192.168.203.128@tcp [ 2326.371855] Key type .llcrypt unregistered [ 2326.373879] Key type ._llcrypt unregistered [ 2331.622473] Key type ._llcrypt registered [ 2331.624808] Key type .llcrypt registered [ 2331.714223] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2337.222486] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules [ 2338.624326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2338.758855] alg: No test for adler32 (adler32-zlib) [ 2339.914405] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2340.193938] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2341.927181] Key type lgssc registered [ 2343.789874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2345.570631] LustreError: 90162:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2363.775439] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2366.693061] LustreError: 90162:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2373.009277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2373.032719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.566346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2378.923869] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2383.848382] LustreError: 91144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.897819] LustreError: 91144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2387.692486] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:24:45 (1786328685) [ 2388.961480] LustreError: 91143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.592632] LustreError: 91175:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2392.600365] LustreError: 91175:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2392.606537] LustreError: 91175:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2392.834365] Lustre: server umount lustre-MDT0000 complete [ 2406.003397] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2409.667571] Key type lgssc unregistered [ 2409.951971] LNet: 92672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.960314] LNetError: 92672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2410.982763] LNet: Removed LNI 192.168.203.128@tcp [ 2412.390374] Key type .llcrypt unregistered [ 2412.396653] Key type ._llcrypt unregistered [ 2431.226546] Key type ._llcrypt registered [ 2431.228447] Key type .llcrypt registered [ 2431.314764] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2433.655986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2433.740978] alg: No test for adler32 (adler32-zlib) [ 2434.835597] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2435.093684] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2436.735339] Key type lgssc registered [ 2437.898595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2445.998837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2446.010903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.668388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.002379] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2457.068953] LustreError: 93892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.098792] LustreError: 93892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2461.305790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.899963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.026445] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2477.800945] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2483.863416] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2490.588878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2491.539558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.578321] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2505.945820] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2558.809484] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:27:36 (1786328856) [ 2561.336955] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2564.025565] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:27:41 (1786328861) [ 2569.184331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2569.190535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.197541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.772215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.775243] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.784594] Lustre: Skipped 1 previous similar message [ 2572.794915] Lustre: Skipped 1 previous similar message [ 2574.938896] Lustre: server umount lustre-MDT0000 complete [ 2577.896498] LustreError: 93891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.002041] LustreError: 93891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2579.285727] LustreError: 93871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786328877 with bad export cookie 11377027828204978148 [ 2579.291333] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2579.295314] LustreError: 93871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2579.633076] Lustre: server umount lustre-MDT0001 complete [ 2594.807869] Lustre: server umount lustre-OST0000 complete [ 2608.436102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.010752] LustreError: 98172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.179700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.241170] LustreError: 98173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.421421] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2619.376945] LustreError: 98172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.430123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.907308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2628.716947] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2637.448475] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2643.549771] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2657.528574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2657.656627] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2657.662144] LustreError: 98161:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2657.676556] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2657.685727] LustreError: 100254:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2657.693682] LustreError: 100254:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2657.701512] LustreError: 100254:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2657.766416] Lustre: server umount lustre-OST0000 complete [ 2657.769605] LustreError: 100254:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2667.729531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2667.840698] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2667.846384] LustreError: 98162:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2667.871230] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2667.877962] LustreError: 100721:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2667.882846] LustreError: 100721:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2667.887667] LustreError: 100721:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2667.948388] Lustre: server umount lustre-OST0000 complete [ 2667.952213] LustreError: 100721:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2678.908857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2679.043244] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2679.047358] Lustre: Found index 0 for lustre-OST0000, updating log [ 2679.053469] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2679.168612] Lustre: lustre-OST0000: new disk, initializing [ 2679.171982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2679.176626] Lustre: 101255:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2679.378938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.518169] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2689.480754] Lustre: Failing over lustre-OST0000 [ 2689.599678] Lustre: server umount lustre-OST0000 complete [ 2694.244636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.271074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.752272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.764402] Lustre: Skipped 2 previous similar messages [ 2702.865595] Lustre: server umount lustre-MDT0000 complete [ 2704.864580] LustreError: 98167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.893476] LustreError: 98167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2706.492304] LustreError: 98153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786329004 with bad export cookie 11377027828204980892 [ 2706.492758] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.554344] LustreError: 98153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2707.056594] Lustre: server umount lustre-MDT0001 complete [ 2716.909764] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:30:14 (1786329014) [ 2734.703902] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2745.631856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.879517] LustreError: 103602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.120447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.742658] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2758.289061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2764.028033] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2767.079770] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2773.302570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.909679] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2780.403568] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2783.155545] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2783.249985] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2783.260325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2788.089369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.175706] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2788.242809] Lustre: lustre-OST0001: new disk, initializing [ 2788.248178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2788.253599] Lustre: 105922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2788.362481] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2788.458716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2788.896102] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2788.899827] Lustre: Skipped 2 previous similar messages [ 2790.966693] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2790.995076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2791.032350] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2791.034040] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2791.043631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2794.200038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2794.213427] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2794.406263] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2796.348598] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2805.316449] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2813.847443] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2819.224273] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [ 2820.078033] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.100273] Lustre: Skipped 3 previous similar messages [ 2829.419903] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [ 2829.428601] Lustre: Skipped 7 previous similar messages [ 2833.375171] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2833.669797] Lustre: server umount lustre-MDT0001 complete [ 2834.567596] LustreError: 103597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.615144] LustreError: 103597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2843.938081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.000883] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2844.004845] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2844.035993] Lustre: 103591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2845.305365] LustreError: 107455:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2845.315405] LustreError: 107455:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2845.330715] LustreError: 103594:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2845.467299] LustreError: 107466:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -17 [ 2845.484503] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2845.484503] [ 2849.782723] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2850.278223] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2861.093416] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:32:38 (1786329158) [ 2881.725117] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2900.719211] Lustre: 109538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2923.185872] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2926.696647] Lustre: 110673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3021.797797] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.799429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3021.811655] Lustre: Skipped 2 previous similar messages [ 3021.820724] Lustre: Skipped 6 previous similar messages [ 3025.635754] Lustre: server umount lustre-MDT0000 complete [ 3026.912821] LustreError: 103597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.932260] LustreError: 103597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3029.171941] LustreError: 103584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786329327 with bad export cookie 11377027828204981893 [ 3029.172313] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.181692] LustreError: 103584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3029.426876] Lustre: server umount lustre-MDT0001 complete [ 3044.623281] Lustre: server umount lustre-OST0000 complete [ 3047.711168] Lustre: 93203:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786329330/real 1786329330] req@ffff895e8c929180 x1873101437909248/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786329346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3047.889627] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.960144] Lustre: Skipped 1 previous similar message [ 3048.895514] Lustre: server umount lustre-OST0001 complete [ 3055.875036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3067.495399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3075.784717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3085.941560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3105.945737] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3116.115300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.131648] Lustre: lustre-MDT0000: reset Object Index mappings [ 3116.592953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.605361] Lustre: Skipped 1 previous similar message [ 3121.270524] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3129.493639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3129.510737] Lustre: lustre-MDT0001: reset Object Index mappings [ 3134.742571] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3138.011216] Lustre: 115774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3145.386741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3153.170695] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3155.441922] LustreError: 116130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.458813] LustreError: 116130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3155.466158] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3161.683200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3162.004669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3167.770248] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3167.771809] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3169.836427] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3179.079695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3182.760151] Lustre: 117652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3248.013307] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:39:06 (1786329546) [ 3275.248209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.255288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3275.260838] Lustre: Skipped 2 previous similar messages [ 3275.278514] Lustre: Skipped 2 previous similar messages [ 3281.171367] Lustre: server umount lustre-MDT0000 complete [ 3284.795857] LustreError: 114610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786329583 with bad export cookie 11377027828205029010 [ 3284.806319] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3299.296972] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3299.537056] Lustre: server umount lustre-MDT0001 complete [ 3310.298547] Lustre: server umount lustre-OST0000 complete [ 3320.748572] Lustre: server umount lustre-OST0001 complete [ 3328.043781] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3337.762561] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3375.980549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.155578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3376.181848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3376.251758] Lustre: lustre-MDT0000: new disk, initializing [ 3376.315498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.320832] Lustre: Skipped 3 previous similar messages [ 3376.333274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3380.929074] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3387.979556] Lustre: server umount lustre-MDT0000 complete [ 3396.236678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.456928] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3396.465577] Lustre: 121927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3396.478483] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3401.126291] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3406.639518] Lustre: server umount lustre-MDT0000 complete [ 3416.062769] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:41:53 (1786329713) [ 3417.822035] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3420.596750] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:41:57 (1786329717) [ 3438.053955] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3447.684293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.586853] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3461.225734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.336923] Lustre: 123979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3461.391572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3461.394901] Lustre: Skipped 1 previous similar message [ 3461.465539] Lustre: lustre-MDT0001: new disk, initializing [ 3461.625437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3461.658702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3465.595472] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3470.755401] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3477.571303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3478.287264] Lustre: lustre-OST0000: new disk, initializing [ 3478.292262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3478.301409] Lustre: 125555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3480.185359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3480.195957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3480.396503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3484.861506] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3496.081556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3496.221537] Lustre: lustre-OST0001: new disk, initializing [ 3496.224892] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3496.230275] Lustre: 126424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3497.417948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3497.435975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3497.792085] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3504.225028] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3514.312355] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3518.531158] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3996.644387] Lustre: 132849:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3996.650162] Lustre: 132849:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3996.654677] Lustre: 132849:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3996.658461] Lustre: 132849:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3996.663451] Lustre: 132849:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3996.668401] Lustre: 132849:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4341.875546] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:57:19 (1786330639) [ 4357.600918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4357.607815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4357.621121] Lustre: Skipped 3 previous similar messages [ 4357.643741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4357.649303] Lustre: Skipped 12 previous similar messages [ 4363.407778] Lustre: server umount lustre-MDT0000 complete [ 4364.263385] LustreError: 127230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.277420] LustreError: 127230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 4366.684350] LustreError: 123970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786330665 with bad export cookie 11377027828205100396 [ 4366.691382] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4366.693446] LustreError: 123970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4369.378166] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4369.379394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4369.387212] Lustre: Skipped 4 previous similar messages [ 4369.391314] Lustre: Skipped 4 previous similar messages [ 4373.154402] Lustre: server umount lustre-MDT0001 complete [ 4382.807605] Lustre: server umount lustre-OST0000 complete [ 4393.755642] Lustre: server umount lustre-OST0001 complete [ 4401.482335] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4410.748936] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4449.141788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4449.320800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4449.346362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4449.417125] Lustre: lustre-MDT0000: new disk, initializing [ 4449.473114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.476799] Lustre: Skipped 5 previous similar messages [ 4449.487687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4454.482689] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4464.086224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.171165] Lustre: 135914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4464.217717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4464.220881] Lustre: Skipped 1 previous similar message [ 4464.290493] Lustre: lustre-MDT0001: new disk, initializing [ 4464.381903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4464.390312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4468.956306] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4478.152169] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4485.544095] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4492.452960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4492.714470] Lustre: lustre-OST0000: new disk, initializing [ 4492.717759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4492.724779] Lustre: 137883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4492.797238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4492.802780] Lustre: Skipped 1 previous similar message [ 4494.827825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4494.860749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4495.059799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4500.037445] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4510.446494] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4521.421904] Lustre: lustre-MDD0000: changelog on [ 4524.230985] Lustre: lustre-MDD0001: changelog on [ 6471.738285] Lustre: lustre-MDD0001: changelog off [ 6473.657650] Lustre: 135919:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6474.436778] Lustre: lustre-MDD0000: changelog off [ 6480.373015] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.387231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6480.397923] Lustre: Skipped 4 previous similar messages [ 6485.487280] LustreError: 139530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6485.504935] LustreError: 139530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6485.601085] Lustre: server umount lustre-OST0000 complete [ 6490.596894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6490.622439] Lustre: Skipped 1 previous similar message [ 6490.633879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6490.645048] Lustre: Skipped 1 previous similar message [ 6495.715856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6495.719678] Lustre: Skipped 1 previous similar message [ 6496.109449] Lustre: server umount lustre-MDT0000 complete [ 6499.936168] LustreError: 136709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786332798 with bad export cookie 11377027828205927089 [ 6499.940417] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6499.947468] LustreError: 136709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6500.362083] Lustre: server umount lustre-MDT0001 complete [ 6511.920580] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6515.431856] Key type lgssc unregistered [ 6515.884609] LNet: 147033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6515.899866] LNetError: 147033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6515.912890] LNet: Removed LNI 192.168.203.128@tcp [ 6516.948170] Key type .llcrypt unregistered [ 6516.955374] Key type ._llcrypt unregistered [ 6539.163422] Key type ._llcrypt registered [ 6539.166731] Key type .llcrypt registered [ 6539.323304] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6554.349894] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6555.740386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6555.960582] alg: No test for adler32 (adler32-zlib) [ 6557.495742] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6557.876967] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6559.517213] Key type lgssc registered [ 6560.531474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6597.047861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6597.072401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.321129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6598.358865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6598.412912] Lustre: lustre-MDT0000: new disk, initializing [ 6598.471095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6598.482931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6602.772511] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6611.581318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.673630] Lustre: 150235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6611.716100] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6611.719684] Lustre: Skipped 1 previous similar message [ 6611.783130] Lustre: lustre-MDT0001: new disk, initializing [ 6611.864548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6611.902366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6611.924520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6616.146323] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6624.197726] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6629.515407] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6637.268302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.560407] Lustre: lustre-OST0000: new disk, initializing [ 6637.564131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6637.573653] Lustre: 152205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6637.713567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6645.461861] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6646.471489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6646.569210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6646.617472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6655.529965] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6661.077908] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_module obdecho/obdecho [ 6662.834474] LustreError: 153209:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6662.839174] LustreError: 153209:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6664.449631] LustreError: 153305:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6678.511324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6678.515694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.524676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6681.060691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.068347] Lustre: Skipped 1 previous similar message [ 6681.072131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6681.075334] Lustre: Skipped 2 previous similar messages [ 6682.571673] Lustre: server umount lustre-MDT0000 complete [ 6686.318444] LustreError: 150227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786332984 with bad export cookie 14641366565423535004 [ 6686.330172] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6686.336313] LustreError: 150227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6686.895541] Lustre: server umount lustre-MDT0001 complete [ 6700.092962] Lustre: server umount lustre-OST0000 complete [ 6720.269442] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6730.438877] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6765.435779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.630613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6765.655695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6765.715208] Lustre: lustre-MDT0000: new disk, initializing [ 6765.777093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6765.794507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6770.358379] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6782.243435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.349377] Lustre: 157018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6782.393452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6782.399086] Lustre: Skipped 1 previous similar message [ 6782.458280] Lustre: lustre-MDT0001: new disk, initializing [ 6782.521805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6782.594189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6782.606964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6788.210290] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6798.995722] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6805.693467] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6813.276647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6813.470440] Lustre: lustre-OST0000: new disk, initializing [ 6813.474981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6813.480042] Lustre: 158989:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6813.557947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6820.228612] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6820.423366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6820.444094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6820.653417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6830.867886] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6836.902774] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6837.222450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6841.279351] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6841.659535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6846.443433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6846.446913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6846.452210] LustreError: Skipped 1 previous similar message [ 6846.453670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6846.468836] Lustre: Skipped 1 previous similar message [ 6849.470425] Lustre: server umount lustre-OST0000 complete [ 6854.112625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6854.121361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6854.138841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6854.147651] Lustre: Skipped 1 previous similar message [ 6859.844137] Lustre: server umount lustre-MDT0000 complete [ 6861.807686] LustreError: 157029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6861.843722] LustreError: 157029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6865.004938] LustreError: 157013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786333163 with bad export cookie 14641366565423535963 [ 6865.044082] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6865.286372] Lustre: server umount lustre-MDT0001 complete [ 6869.290687] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:39:27 (1786333167) [ 6871.694671] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6873.866920] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 23:39:31 (1786333171) [ 6894.224129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.559598] LustreError: 161962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6894.632569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.243544] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6899.681808] LustreError: 161963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6903.780235] LustreError: 161962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6908.098940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.857304] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6923.202634] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6929.113344] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6938.345664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6938.627453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6938.642449] Lustre: Skipped 1 previous similar message [ 6945.565043] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6954.062320] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6960.786199] Lustre: *** cfs_fail_loc=718, val=0*** [ 6963.121425] Lustre: server umount lustre-MDT0000 complete [ 6963.176688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6963.184175] LustreError: 164319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6963.189116] Lustre: Skipped 2 previous similar messages [ 6963.208812] LustreError: 164319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6966.559942] LustreError: 161942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786333264 with bad export cookie 14641366565423536845 [ 6966.565727] Lustre: *** cfs_fail_loc=718, val=0*** [ 6966.567923] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6966.581485] LustreError: 161942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6966.583742] Lustre: Skipped 3 previous similar messages [ 6966.857880] Lustre: server umount lustre-MDT0001 complete [ 6978.661629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6983.711219] Lustre: 148463:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786333266/real 1786333266] req@00000000ad2161f7 x1873105760927104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786333282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6983.727505] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.733622] Lustre: Skipped 1 previous similar message [ 6986.847174] Lustre: 148460:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786333269/real 1786333269] req@000000008ff088bf x1873105760927232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786333285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6989.535429] Lustre: 148461:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786333271/real 1786333271] req@00000000bb154f29 x1873105760927360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786333287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6990.968590] LustreError: 148459:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008ff088bf x1873105760928256/t0(0) o250->MGC192.168.203.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6991.074638] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcb3095516659ddf2 [ 6991.086799] Lustre: MGC192.168.203.128@tcp: Connection restored to 0@lo (at 0@lo) [ 6991.274429] Lustre: *** cfs_fail_loc=723, val=0*** [ 6991.280015] Lustre: Skipped 1 previous similar message [ 6991.362210] LustreError: 165321:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 6991.365452] LustreError: 165321:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -122 [ 6991.370102] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6991.370102] [ 6991.505877] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6991.528650] LustreError: 165290:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 6991.588866] LustreError: 165290:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -122 [ 6991.608474] LustreError: 165290:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6991.627859] LustreError: 165290:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6991.791760] Lustre: server umount lustre-MDT0000 complete [ 6991.813231] LustreError: 165290:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 6992.675747] Lustre: 148463:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786333274/real 1786333274] req@000000007e825fc7 x1873105760927616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786333290 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6996.896920] Lustre: 148463:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786333279/real 1786333279] req@00000000799cba4a x1873105760927744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786333295 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7001.081784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7001.180281] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7001.191495] LustreError: 161954:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 7001.381416] LustreError: 163794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7001.393443] LustreError: 163794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7001.518197] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7003.561888] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7006.088993] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7006.120545] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7015.126825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.290141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 7015.381112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7015.392639] Lustre: Skipped 1 previous similar message [ 7015.457518] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7017.792359] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7019.670976] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7020.530505] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7020.650174] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7028.293508] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7030.444928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7036.424217] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7037.946816] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7041.817705] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 7056.366729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7056.387205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7056.397342] Lustre: Skipped 3 previous similar messages [ 7058.830143] Lustre: server umount lustre-OST0000 complete [ 7066.593238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7066.608156] Lustre: Skipped 1 previous similar message [ 7077.343127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7077.534452] Lustre: server umount lustre-MDT0000 complete [ 7080.925508] LustreError: 161943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786333379 with bad export cookie 14641366565423537797 [ 7080.940506] LustreError: 161943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7080.942479] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7081.292329] Lustre: server umount lustre-MDT0001 complete [ 7092.403949] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 7094.546445] Key type lgssc unregistered [ 7094.801982] LNet: 168800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7094.806677] LNetError: 168800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7094.819775] LNet: Removed LNI 192.168.203.128@tcp [ 7095.548485] Key type .llcrypt unregistered [ 7095.550559] Key type ._llcrypt unregistered [ 7126.866120] Key type ._llcrypt registered [ 7126.868747] Key type .llcrypt registered [ 7126.956756] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7128.908300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7128.965645] alg: No test for adler32 (adler32-zlib) [ 7130.004164] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7130.342396] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 7131.999155] Key type lgssc registered [ 7132.801979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7139.955159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7139.966381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7141.297076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7146.426888] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7151.586332] LustreError: 170669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7151.614128] LustreError: 170669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7156.357205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.900658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7161.713512] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7170.144184] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7175.485885] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7182.168089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7182.428928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7186.548920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7188.551514] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7195.403637] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7207.461437] Lustre: *** cfs_fail_loc=718, val=0*** [ 7208.480203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7208.507758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7208.523216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7209.078524] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.28@tcp (stopping) [ 7212.514103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7212.515412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.762176] Lustre: server umount lustre-MDT0000 complete [ 7214.186990] LustreError: 170666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7217.651885] LustreError: 170669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.057925] LustreError: 170669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7218.773992] LustreError: 170652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786333517 with bad export cookie 12537330937380472724 [ 7218.775906] Lustre: *** cfs_fail_loc=718, val=0*** [ 7218.781771] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7218.792235] LustreError: 170652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7218.812452] Lustre: Skipped 3 previous similar messages [ 7219.185635] Lustre: server umount lustre-MDT0001 complete [ 7231.827186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.160441] Lustre: 169983:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786333521/real 1786333521] req@0000000050a43615 x1873106360628992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786333537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7240.199708] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.206991] Lustre: Skipped 1 previous similar message [ 7244.319180] Lustre: 169980:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786333526/real 1786333526] req@000000007ece2380 x1873106360629248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786333542 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7244.324342] LustreError: 169979:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b5e49875 x1873106360630144/t0(0) o250->MGC192.168.203.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7244.418469] Lustre: 169980:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7244.702350] Lustre: *** cfs_fail_loc=727, val=0*** [ 7244.759614] LustreError: 174076:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-57) [ 7244.764720] LustreError: 174076:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -57 [ 7244.769595] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7244.769595] [ 7244.774823] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -57. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7244.782494] LustreError: 174047:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -57 [ 7244.786327] LustreError: 174047:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -57 [ 7244.791569] LustreError: 174047:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7244.796889] LustreError: 174047:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7244.888929] Lustre: server umount lustre-MDT0000 complete [ 7244.903063] LustreError: 174047:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -57 [ 7245.343569] Lustre: 169981:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786333527/real 1786333527] req@00000000efcf6e3e x1873106360629504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786333543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7249.893274] Lustre: 169982:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786333532/real 1786333532] req@000000001dbb0eb7 x1873106360629632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786333548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7255.018134] Lustre: 169983:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786333537/real 1786333537] req@0000000081490693 x1873106360629760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786333553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7255.821909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.936291] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7255.948220] LustreError: 170662:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 7256.205487] LustreError: 172499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7256.286778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.336686] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7261.165616] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7261.370703] LustreError: 174582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7263.475644] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7267.924534] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7270.141620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7270.299408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 7270.376103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7270.410383] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7273.629142] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7275.251198] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7275.505856] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7275.704320] Lustre: lustre-MDT0001: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 7275.776416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7283.927657] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7285.424377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7291.218409] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7293.414174] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7297.329968] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 7311.336536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.352281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7311.357451] Lustre: Skipped 2 previous similar messages [ 7315.166890] Lustre: server umount lustre-OST0000 complete [ 7321.574502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7321.577532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7321.587111] Lustre: Skipped 2 previous similar messages [ 7321.612700] Lustre: Skipped 2 previous similar messages [ 7325.174963] Lustre: server umount lustre-MDT0000 complete [ 7326.690122] LustreError: 174582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.732127] LustreError: 174582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7328.680639] LustreError: 170651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786333627 with bad export cookie 12537330937380473921 [ 7328.690191] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7328.691894] LustreError: 170651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7329.083525] Lustre: server umount lustre-MDT0001 complete [ 7341.341420] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 7343.660638] Key type lgssc unregistered [ 7343.971772] LNet: 177651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7344.041124] LNetError: 177651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7344.066319] LNet: Removed LNI 192.168.203.128@tcp [ 7344.716180] Key type .llcrypt unregistered [ 7344.717961] Key type ._llcrypt unregistered [ 7366.419333] Key type ._llcrypt registered [ 7366.422063] Key type .llcrypt registered [ 7366.513378] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7368.514155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7368.604488] alg: No test for adler32 (adler32-zlib) [ 7369.696865] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7370.019615] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 7371.695242] Key type lgssc registered [ 7372.945951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7379.967678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7380.005845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7381.413046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7385.812876] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7390.701751] LustreError: 178967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.719263] LustreError: 178967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7394.250661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7394.499814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7398.745950] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7406.755791] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7412.457633] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7419.305612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7419.367410] LustreError: 180789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7419.836638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7426.036834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7426.521922] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7433.750978] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7445.984311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7445.995674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7446.018281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7446.497917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7446.654606] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.28@tcp (stopping) [ 7446.670111] Lustre: Skipped 3 previous similar messages [ 7447.427722] Lustre: server umount lustre-MDT0000 complete [ 7450.757804] LustreError: 178948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786333749 with bad export cookie 15439653692058829554 [ 7450.776795] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7450.781549] LustreError: 178948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7451.618708] LustreError: 180209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7451.619883] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7451.644847] Lustre: Skipped 1 previous similar message [ 7451.648592] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7455.654202] LustreError: 178964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7455.654812] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7455.668509] LustreError: 178964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7455.688652] Lustre: Skipped 1 previous similar message [ 7457.148478] Lustre: server umount lustre-MDT0001 complete [ 7463.244161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7475.357784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.539587] LustreError: 182808:0:(llog.c:1655:llog_backup()) MGC192.168.203.128@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7475.545659] Lustre: 182808:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.128@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7481.376645] LustreError: 182813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7481.394325] LustreError: 182813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7491.688569] LustreError: 182813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.708141] LustreError: 182813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7508.074020] LustreError: 182815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.122419] LustreError: 182815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 7516.135869] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7516.141296] Lustre: Skipped 1 previous similar message [ 7536.607308] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7536.618702] Lustre: 182808:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7542.257124] LustreError: 182815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7542.276432] LustreError: 182815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 7577.602467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7582.375935] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7582.695622] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7593.121863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.237589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7596.326657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 7601.203932] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7601.642803] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7609.093552] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7611.701452] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7618.148775] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7620.170484] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7642.597251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7642.600794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7642.625582] Lustre: Skipped 2 previous similar messages [ 7642.626532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7642.631521] Lustre: Skipped 2 previous similar messages [ 7643.300993] Lustre: server umount lustre-MDT0000 complete [ 7647.260740] LustreError: 181748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786333945 with bad export cookie 15439653692058836225 [ 7647.264817] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7647.269270] LustreError: 181748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7647.757886] LustreError: 182815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.761656] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.814676] LustreError: 182815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 40 previous similar messages [ 7648.002850] Lustre: server umount lustre-MDT0001 complete [ 7654.576425] Lustre: server umount lustre-OST0000 complete [ 7678.436227] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 7689.421115] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7726.897799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7727.238123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7727.279441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7727.341948] Lustre: lustre-MDT0000: new disk, initializing [ 7727.405057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7727.444614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7732.203281] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7743.318047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.377324] Lustre: 187912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7743.431532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7743.434693] Lustre: Skipped 1 previous similar message [ 7743.493510] Lustre: lustre-MDT0001: new disk, initializing [ 7743.574668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7743.606479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7743.619076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7748.706859] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7759.139214] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7764.899147] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7772.338947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7772.526391] Lustre: lustre-OST0000: new disk, initializing [ 7772.529354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7772.534153] Lustre: 189880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7778.694786] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7780.918476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7780.928612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7781.139819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7788.202664] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7792.618461] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7792.856390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7797.529272] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7797.685335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7801.826266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7801.826371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7801.832168] LustreError: Skipped 1 previous similar message [ 7801.917373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7801.952308] Lustre: Skipped 1 previous similar message [ 7806.423911] Lustre: server umount lustre-OST0000 complete [ 7817.673670] Lustre: server umount lustre-MDT0000 complete [ 7821.789645] LustreError: 187904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786334120 with bad export cookie 15439653692058853284 [ 7821.799875] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7821.804877] LustreError: 187904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7822.294848] Lustre: server umount lustre-MDT0001 complete [ 7831.392895] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:55:29 (1786334129) [ 7838.769087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7839.161487] LustreError: 192290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7839.187569] LustreError: 192290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7839.290349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7839.296359] Lustre: Skipped 1 previous similar message [ 7843.594328] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7852.996183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7857.690341] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7864.953379] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7869.826289] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7876.913645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7883.474848] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7890.512820] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7901.748983] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:56:39 (1786334199) [ 7908.835442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7908.839127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7908.844928] Lustre: Skipped 3 previous similar messages [ 7908.848381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7908.851594] Lustre: Skipped 4 previous similar messages [ 7912.058627] Lustre: server umount lustre-OST0000 complete [ 7921.906741] Lustre: server umount lustre-MDT0000 complete [ 7925.499245] LustreError: 192272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786334223 with bad export cookie 15439653692058854159 [ 7925.512774] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7925.521556] LustreError: 192272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7925.741622] Lustre: server umount lustre-MDT0001 complete [ 7937.130983] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 7939.893942] Key type lgssc unregistered [ 7940.123765] LNet: 196020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7940.128727] LNetError: 196020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7940.140651] LNet: Removed LNI 192.168.203.128@tcp [ 7940.739921] Key type .llcrypt unregistered [ 7940.741774] Key type ._llcrypt unregistered [ 7959.542623] Key type ._llcrypt registered [ 7959.544707] Key type .llcrypt registered [ 7959.623628] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7961.394197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7961.427774] alg: No test for adler32 (adler32-zlib) [ 7962.657891] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7963.097274] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 7964.752757] Key type lgssc registered [ 7965.639546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7972.365329] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7972.377445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.871569] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7992.997772] LustreError: 197263:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 7998.112749] LustreError: 197263:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7998.127068] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7998.131462] LustreError: MGC192.168.203.128@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7998.137131] LustreError: 197263:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7998.142331] LustreError: 197263:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7998.146237] LustreError: 197263:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7998.186614] Lustre: server umount lustre-OST0000 complete [ 7998.190357] LustreError: 197263:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8006.087466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8007.666428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8011.450260] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8017.915495] LustreError: 197762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8018.065604] LustreError: 197762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8020.698637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.227200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8025.403102] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8032.624935] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8039.145555] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8047.227822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8047.607407] LustreError: 199578:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8047.623852] Lustre: 199578:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.128@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8047.928797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8054.472760] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8061.896204] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8066.451314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8066.463933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8066.486300] Lustre: Skipped 2 previous similar messages [ 8066.530801] Lustre: Skipped 2 previous similar messages [ 8070.762186] Lustre: server umount lustre-MDT0000 complete [ 8071.667422] LustreError: 199002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8071.676650] LustreError: 199002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8074.342414] LustreError: 197743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786334372 with bad export cookie 15123267821644218495 [ 8074.347288] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8074.356724] LustreError: 197743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8074.743526] LustreError: 198492:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000471e3dc3 x1873107234091136/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8074.982674] Lustre: server umount lustre-MDT0001 complete [ 8089.263418] Lustre: server umount lustre-OST0000 complete [ 8098.783236] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 8109.742773] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8147.218222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8147.387240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8147.408925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8147.482316] Lustre: lustre-MDT0000: new disk, initializing [ 8147.545575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8147.573192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8151.816706] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8163.745492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8163.886893] Lustre: 203106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8163.930876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8163.934422] Lustre: Skipped 1 previous similar message [ 8163.991536] Lustre: lustre-MDT0001: new disk, initializing [ 8164.145707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8164.235459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8164.248482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8168.602619] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8178.960364] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8185.422834] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8192.007085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.526203] Lustre: lustre-OST0000: new disk, initializing [ 8192.529732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8192.536312] Lustre: 205089:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8192.608350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8198.498308] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8201.337200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8201.362167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8201.647638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8208.523090] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8213.900820] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8214.073466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8218.692938] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8218.939684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8222.179343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8222.180128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8222.191193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8222.193301] Lustre: Skipped 1 previous similar message [ 8222.206413] Lustre: Skipped 1 previous similar message [ 8227.325105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8227.333808] Lustre: Skipped 1 previous similar message [ 8227.577767] Lustre: server umount lustre-OST0000 complete [ 8230.880406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8230.928201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.956026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8237.536542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8237.542682] Lustre: Skipped 3 previous similar messages [ 8245.215261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8245.406426] Lustre: server umount lustre-MDT0000 complete [ 8247.780421] LustreError: 204260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8247.794235] LustreError: 204260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8249.017431] LustreError: 205088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786334547 with bad export cookie 15123267821644219223 [ 8249.029592] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8249.039099] LustreError: 205088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8249.438895] Lustre: server umount lustre-MDT0001 complete [ 8261.015952] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:02:38 (1786334558) [ 8279.823806] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 8282.133718] Key type lgssc unregistered [ 8282.482739] LNet: 208040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8282.493164] LNetError: 208040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8282.504958] LNet: Removed LNI 192.168.203.128@tcp [ 8284.157980] Key type .llcrypt unregistered [ 8284.160526] Key type ._llcrypt unregistered [ 8303.419499] Key type ._llcrypt registered [ 8303.426784] Key type .llcrypt registered [ 8303.614689] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8305.919870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8306.097771] alg: No test for adler32 (adler32-zlib) [ 8308.904688] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 8309.503120] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 8311.287189] Key type lgssc registered [ 8312.469515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8321.510825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8321.541459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8322.990593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8329.099518] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8333.282336] LustreError: 209258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8333.310477] LustreError: 209258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8337.865977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8338.115754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8343.168270] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8351.777431] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8358.024801] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8365.473265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8365.604655] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8365.607497] LustreError: 211122:0:(llog.c:1696:llog_backup()) MGC192.168.203.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8365.613288] Lustre: 211122:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.128@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8365.712218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8371.849720] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8379.755454] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8383.457940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8383.461977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8383.472988] Lustre: Skipped 1 previous similar message [ 8383.488482] Lustre: Skipped 2 previous similar messages [ 8384.552412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8387.930481] Lustre: server umount lustre-MDT0000 complete [ 8389.605725] LustreError: 209255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8389.616643] LustreError: 209255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8392.180390] LustreError: 209239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786334690 with bad export cookie 17265824171858936166 [ 8392.195602] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8392.823879] LustreError: 209987:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000687c5d96 x1873107596897792/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8393.113545] Lustre: server umount lustre-MDT0001 complete [ 8407.541436] Lustre: server umount lustre-OST0000 complete [ 8417.114411] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 8430.211582] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8467.099787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8467.286546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8467.332651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8467.380556] Lustre: lustre-MDT0000: new disk, initializing [ 8467.446611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8467.465515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8473.085609] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8483.472572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8483.544989] Lustre: 214650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8483.591087] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8483.593436] Lustre: Skipped 1 previous similar message [ 8483.596601] LustreError: 215430:0:(llog.c:1696:llog_backup()) MGC192.168.203.128@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8483.601662] LustreError: 215430:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8483.605903] Lustre: 215430:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.128@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8483.620300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8483.623856] Lustre: Skipped 1 previous similar message [ 8483.681906] Lustre: lustre-MDT0001: new disk, initializing [ 8483.755626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8483.794262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8483.807882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8487.862799] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8498.393306] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8505.315703] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8512.124940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8512.602758] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8512.606865] Lustre: Skipped 1 previous similar message [ 8512.611409] LustreError: 216591:0:(llog.c:1696:llog_backup()) MGC192.168.203.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8512.622204] LustreError: 216591:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8512.636248] Lustre: 216591:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.128@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8512.648418] Lustre: 216591:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8512.682815] Lustre: lustre-OST0000: new disk, initializing [ 8512.687274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8512.697595] Lustre: 216615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8512.820387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8518.358120] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8519.857211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8519.913323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8520.490236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8528.854368] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8533.549214] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8533.852734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8538.242537] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8538.517822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8540.641686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8540.652116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8540.683646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8540.695177] Lustre: Skipped 2 previous similar messages [ 8540.762090] Lustre: Skipped 3 previous similar messages [ 8545.762905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8545.767389] Lustre: Skipped 1 previous similar message [ 8546.590461] Lustre: server umount lustre-OST0000 complete [ 8550.375186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8550.388893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.414120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8559.012753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8559.016255] Lustre: Skipped 4 previous similar messages [ 8564.192407] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8564.323454] Lustre: server umount lustre-MDT0000 complete [ 8568.755654] LustreError: 214641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786334867 with bad export cookie 17265824171858936887 [ 8568.772371] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8569.169196] Lustre: server umount lustre-MDT0001 complete [ 8578.792222] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:07:56 (1786334876) [ 8595.389478] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8607.491290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.772490] LustreError: 219744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8607.833972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8612.849056] LustreError: 219745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.071405] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8617.953758] LustreError: 219744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8621.864902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8627.125728] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8629.803438] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8636.514682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8642.596932] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8651.163571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8651.299427] Lustre: lustre-OST0001: new disk, initializing [ 8651.302639] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8651.309523] Lustre: 222052:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8651.391527] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8651.400404] Lustre: Skipped 2 previous similar messages [ 8656.918241] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8659.570253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8659.579742] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8659.698280] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8668.854810] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8676.584566] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8678.588947] Lustre: DEBUG MARKER: ADD OST3 [ 8684.149990] loop: module loaded [ 8684.271233] loop0: detected capacity change from 0 to 204800000 [ 8684.541509] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8684.550912] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8684.557626] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8690.353222] loop0: detected capacity change from 0 to 204800000 [ 8692.460362] Lustre: DEBUG MARKER: STOP OST3 [ 8699.471233] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8699.741331] Lustre: lustre-OST0003: new disk, initializing [ 8699.745344] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8699.752984] Lustre: 224014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8701.287521] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8701.570785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8701.588854] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8701.653365] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8703.328112] LustreError: 224087:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8705.993038] Lustre: Failing over lustre-OST0003 [ 8706.179485] LustreError: 222582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8706.260480] Lustre: server umount lustre-OST0003 complete [ 8706.545261] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8706.561721] Lustre: Skipped 1 previous similar message [ 8709.975899] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8720.344732] Lustre: DEBUG MARKER: START OST3 again [ 8726.429037] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8726.778122] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8726.793215] Lustre: Skipped 1 previous similar message [ 8726.828508] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8728.311805] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8728.813258] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8728.815774] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8732.942734] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8753.290815] Lustre: Failing over lustre-OST0003 [ 8753.383313] Lustre: server umount lustre-OST0003 complete [ 8754.656718] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8754.666322] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.685180] LustreError: 221233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.720675] LustreError: 221233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8770.531804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8770.542701] LustreError: Skipped 1 previous similar message [ 8770.550743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8770.569038] Lustre: Skipped 1 previous similar message [ 8770.575798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8770.583166] Lustre: Skipped 2 previous similar messages [ 8784.352057] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8784.493213] Lustre: server umount lustre-MDT0000 complete [ 8787.695787] LustreError: 219726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786335086 with bad export cookie 17265824171858937769 [ 8787.700591] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8787.702438] LustreError: 219726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8788.448627] LustreError: 222180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8788.450572] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.462086] LustreError: 222180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8788.470245] Lustre: Skipped 4 previous similar messages [ 8794.106063] Lustre: server umount lustre-MDT0001 complete [ 8797.620680] Lustre: server umount lustre-OST0000 complete [ 8801.281498] Lustre: server umount lustre-OST0001 complete [ 8808.031383] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 8818.653565] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8857.912489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8858.108906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8858.130802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8858.185497] Lustre: lustre-MDT0000: new disk, initializing [ 8858.254299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8858.270368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8862.679814] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8873.185215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8873.272520] Lustre: 229512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8873.505919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8877.904393] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8886.488807] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8892.353784] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8898.608743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8898.911761] Lustre: lustre-OST0000: new disk, initializing [ 8898.914906] Lustre: Skipped 1 previous similar message [ 8898.919932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8898.924340] Lustre: Skipped 2 previous similar messages [ 8898.929165] Lustre: 231480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8900.738018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8900.751807] Lustre: Skipped 1 previous similar message [ 8900.762427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8900.876132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8905.412185] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8914.388256] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8918.794981] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8918.988751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8923.054842] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8923.255343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8926.688830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8926.691171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8926.692676] LustreError: Skipped 1 previous similar message [ 8926.693491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8926.693495] Lustre: Skipped 16 previous similar messages [ 8926.731407] Lustre: Skipped 1 previous similar message [ 8931.174290] Lustre: server umount lustre-OST0000 complete [ 8934.881138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8949.215156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8949.450179] Lustre: server umount lustre-MDT0000 complete [ 8952.291744] LustreError: 229521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8952.301532] LustreError: 229521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8952.949800] LustreError: 229502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786335251 with bad export cookie 17265824171858940065 [ 8952.955589] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8952.966231] LustreError: 229502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8953.468818] Lustre: server umount lustre-MDT0001 complete [ 8956.870425] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:14:14 (1786335254) [ 8976.089465] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 8986.144888] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 9022.113297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9022.344317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9022.401601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9022.507128] Lustre: lustre-MDT0000: new disk, initializing [ 9022.597301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9026.794642] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9038.636642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.792381] Lustre: 236089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9039.125350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9043.886825] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9053.811387] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9060.072651] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9066.498190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9066.819220] Lustre: 238059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9069.207455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9073.615391] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9082.623702] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9088.114223] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9088.518955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9092.972641] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9093.129260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9095.137396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9095.144159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9095.151592] Lustre: Skipped 2 previous similar messages [ 9095.157845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9095.161552] Lustre: Skipped 8 previous similar messages [ 9101.488498] Lustre: server umount lustre-OST0000 complete [ 9111.250083] Lustre: server umount lustre-MDT0000 complete [ 9114.407681] LustreError: 236080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786335412 with bad export cookie 17265824171858940947 [ 9114.426386] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9114.433348] LustreError: 236080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9114.846980] Lustre: server umount lustre-MDT0001 complete [ 9121.847876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9122.144406] LustreError: 240277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9122.202436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9122.207787] Lustre: Skipped 5 previous similar messages [ 9126.531455] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9134.430536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9138.701933] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9145.833548] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9151.375460] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9159.057489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9165.555619] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9172.910238] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9183.200541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9183.252747] LustreError: Skipped 2 previous similar messages [ 9188.753314] Lustre: server umount lustre-OST0000 complete [ 9198.714630] Lustre: server umount lustre-MDT0000 complete [ 9202.362793] LustreError: 240258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786335500 with bad export cookie 17265824171858941829 [ 9202.364555] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9202.370385] LustreError: 240258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9202.603546] Lustre: server umount lustre-MDT0001 complete [ 9314.336381] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 9316.590202] Key type lgssc unregistered [ 9316.925653] LNet: 243943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9316.929995] LNetError: 243943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9316.938953] LNet: Removed LNI 192.168.203.128@tcp [ 9317.619139] Key type .llcrypt unregistered [ 9317.623439] Key type ._llcrypt unregistered [ 9344.871321] Key type ._llcrypt registered [ 9344.873149] Key type .llcrypt registered [ 9344.956746] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 9357.803925] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 9358.988792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9359.041528] alg: No test for adler32 (adler32-zlib) [ 9360.077680] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 9360.344449] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 9361.999153] Key type lgssc registered [ 9363.382666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9395.763312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9395.835848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9397.014291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9397.063919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9397.111706] Lustre: lustre-MDT0000: new disk, initializing [ 9397.157243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9397.185370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9401.968858] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9412.256708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9412.328377] Lustre: 247276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9412.364230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9412.367297] Lustre: Skipped 1 previous similar message [ 9412.429525] Lustre: lustre-MDT0001: new disk, initializing [ 9412.514904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9412.541366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9412.556205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9416.806301] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9425.326400] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9431.221816] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9436.813276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9437.070531] Lustre: lustre-OST0000: new disk, initializing [ 9437.076839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9437.082858] Lustre: 249244:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9437.194436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9442.792897] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9444.405953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9444.411582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9444.489665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9450.851752] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9455.817865] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9456.045142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9460.423650] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9460.638226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9465.313306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9465.316569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9465.318504] LustreError: Skipped 1 previous similar message [ 9465.319814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9465.326867] Lustre: Skipped 1 previous similar message [ 9468.599246] Lustre: server umount lustre-OST0000 complete [ 9474.528661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9474.532297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9474.539765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9474.543869] Lustre: Skipped 1 previous similar message [ 9475.557351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9475.559734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9475.574908] Lustre: Skipped 1 previous similar message [ 9480.673501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9480.684853] Lustre: Skipped 1 previous similar message [ 9484.771902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9484.785789] Lustre: Skipped 1 previous similar message [ 9486.303138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9486.435906] Lustre: server umount lustre-MDT0000 complete [ 9490.096329] LustreError: 247266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786335788 with bad export cookie 14542458760853312923 [ 9490.097523] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9490.103171] LustreError: 247266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9490.460961] Lustre: server umount lustre-MDT0001 complete [ 9507.808420] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 9517.053130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9517.315111] LustreError: 252178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.434748] LustreError: 252178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9517.477828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9522.418948] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9522.672992] LustreError: 252179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9526.786515] LustreError: 252178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.465816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9531.688361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9535.720459] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9538.344391] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9544.582876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9544.974684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9550.517210] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9558.105883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9558.417123] Lustre: lustre-OST0001: new disk, initializing [ 9558.422621] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9558.443553] Lustre: 254485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9564.650387] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9566.774487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9566.785928] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9566.839402] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9576.118764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9585.686209] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9598.353326] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:24:56 (1786335896) [ 9602.576568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9602.586649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9602.594757] Lustre: Skipped 3 previous similar messages [ 9602.600557] Lustre: Skipped 2 previous similar messages [ 9606.727922] Lustre: server umount lustre-MDT0000 complete [ 9607.659897] LustreError: 253232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9607.672022] LustreError: 253232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9609.924660] LustreError: 255255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786335908 with bad export cookie 14542458760853313805 [ 9609.946142] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9610.390150] Lustre: server umount lustre-MDT0001 complete [ 9624.734187] Lustre: server umount lustre-OST0000 complete [ 9637.967875] Lustre: server umount lustre-OST0001 complete [ 9643.571221] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 9652.085846] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 9683.519923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9683.706726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9683.724495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9683.792835] Lustre: lustre-MDT0000: new disk, initializing [ 9683.854096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9683.859946] Lustre: Skipped 1 previous similar message [ 9683.872913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9687.970703] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9698.577186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.692918] Lustre: 258432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9698.740600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9698.744833] Lustre: Skipped 1 previous similar message [ 9698.822777] Lustre: lustre-MDT0001: new disk, initializing [ 9698.952372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9698.961705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9703.473067] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9713.014394] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9718.363994] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9724.830924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9725.023956] Lustre: lustre-OST0000: new disk, initializing [ 9725.027513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9725.033245] Lustre: 260398:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9725.114677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9725.126458] Lustre: Skipped 1 previous similar message [ 9726.308587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9726.313143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9726.392903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9730.644770] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9739.279752] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9744.125084] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9744.290676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9749.360240] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9749.633501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9752.036709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9752.037824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.041355] LustreError: Skipped 1 previous similar message [ 9752.042307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9752.042311] Lustre: Skipped 1 previous similar message [ 9752.064211] Lustre: Skipped 1 previous similar message [ 9757.671684] Lustre: server umount lustre-OST0000 complete [ 9762.273581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9762.286836] Lustre: Skipped 1 previous similar message [ 9768.338915] Lustre: server umount lustre-MDT0000 complete [ 9772.383608] LustreError: 258423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786336070 with bad export cookie 14542458760853315947 [ 9772.398966] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9772.405303] LustreError: 258423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9772.841686] Lustre: server umount lustre-MDT0001 complete [ 9781.555516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9781.870645] LustreError: 262617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9786.457367] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9792.503236] LustreError: 262618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.534504] LustreError: 262618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9794.087344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9794.366836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9794.384352] Lustre: Skipped 1 previous similar message [ 9798.539493] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9805.606605] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9811.021672] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9816.263683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9822.293113] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 9828.792554] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9978.851335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9978.857497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9978.871920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9978.874602] Lustre: Skipped 7 previous similar messages [ 9981.631848] Lustre: server umount lustre-OST0000 complete [ 9999.426434] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9999.529482] Lustre: server umount lustre-MDT0000 complete [10000.873659] LustreError: 262618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10000.928069] LustreError: 262618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10003.362503] LustreError: 262597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786336301 with bad export cookie 14542458760853316822 [10003.377311] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10003.379533] LustreError: 262597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10003.622282] Lustre: server umount lustre-MDT0001 complete [10098.990165] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [10101.016994] Key type lgssc unregistered [10101.213890] LNet: 266228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10101.220803] LNetError: 266228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10101.233822] LNet: Removed LNI 192.168.203.128@tcp [10101.903282] Key type .llcrypt unregistered [10101.904867] Key type ._llcrypt unregistered [10128.225965] Key type ._llcrypt registered [10128.227839] Key type .llcrypt registered [10128.309391] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [10144.979771] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10146.522978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10146.653194] alg: No test for adler32 (adler32-zlib) [10147.724972] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [10147.930470] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [10149.719166] Key type lgssc registered [10151.066244] Lustre: Echo OBD driver; http://www.lustre.org/ [10199.193878] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10210.706870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10210.725702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10212.150755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10212.215203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10212.265926] Lustre: lustre-MDT0000: new disk, initializing [10212.323740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10212.339359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10216.607340] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10227.914932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10228.051792] Lustre: 270300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10228.179510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10228.183248] Lustre: Skipped 1 previous similar message [10228.242493] Lustre: lustre-MDT0001: new disk, initializing [10228.473769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10228.520492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10228.558753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10233.355500] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10238.287615] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10243.802607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10244.032805] Lustre: lustre-OST0000: new disk, initializing [10244.036964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10244.048486] Lustre: 271931:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10244.150225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10244.264454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10244.280368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10244.559871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10250.546885] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10261.328351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10261.433661] Lustre: lustre-OST0001: new disk, initializing [10261.437748] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10261.444388] Lustre: 272802:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10261.542221] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10268.453094] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10268.716135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10268.728421] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10268.799237] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10279.945272] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10289.202450] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10300.898601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10300.908225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10300.923065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10304.487058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10304.491993] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10304.517506] Lustre: Skipped 2 previous similar messages [10306.939899] Lustre: server umount lustre-MDT0000 complete [10310.482142] LustreError: 270293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786336608 with bad export cookie 15506960752624604413 [10310.487542] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10310.488486] LustreError: 270293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10310.682458] Lustre: server umount lustre-MDT0001 complete [10326.661701] Lustre: server umount lustre-OST0000 complete [10341.778027] Lustre: server umount lustre-OST0001 complete [10353.726167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10363.072454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10372.156468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10382.212454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10402.561431] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10413.437904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10413.449883] Lustre: lustre-MDT0000: reset Object Index mappings [10413.753835] LustreError: 277808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.768930] LustreError: 277808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10413.829428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10418.275270] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10419.172839] LustreError: 277809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.292426] LustreError: 277808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10426.437902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10426.451936] Lustre: lustre-MDT0001: reset Object Index mappings [10426.824267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10431.240426] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10434.026349] Lustre: 278952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10441.003519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10448.114931] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10451.697465] LustreError: 279306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.748327] LustreError: 279306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10456.719720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10456.992456] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10457.016886] Lustre: Skipped 1 previous similar message [10460.082785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10460.117561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10463.833735] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10473.281494] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10477.063161] Lustre: 280825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10483.168671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10483.176689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10483.183546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10483.187024] Lustre: Skipped 3 previous similar messages [10483.823985] Lustre: server umount lustre-MDT0000 complete [10485.736554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10485.738386] LustreError: 277809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10485.744904] Lustre: Skipped 1 previous similar message [10485.762876] LustreError: 277809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10487.429853] LustreError: 280827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786336785 with bad export cookie 15506960752624611203 [10487.438994] LustreError: 280827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10487.440842] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10488.287836] Lustre: server umount lustre-MDT0001 complete [10501.912886] Lustre: server umount lustre-OST0000 complete [10516.232237] Lustre: server umount lustre-OST0001 complete [10537.560444] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:40:35 (1786336835) [10552.595915] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [10563.416610] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10600.034890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.275145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10600.295451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10600.354935] Lustre: lustre-MDT0000: new disk, initializing [10600.421686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10600.440235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10604.939896] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10615.883922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.007119] Lustre: 285078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10616.049663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10616.052979] Lustre: Skipped 1 previous similar message [10616.105477] Lustre: lustre-MDT0001: new disk, initializing [10616.197944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10616.207834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10620.836603] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10630.587518] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10636.625224] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10643.488356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10643.724500] Lustre: lustre-OST0000: new disk, initializing [10643.727950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10643.733474] Lustre: 287049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10645.070107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10645.084155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10645.180667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10649.491941] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10659.581495] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10664.602889] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10664.789906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10669.567818] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10670.144556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10676.192400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10676.197259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10676.202811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10676.212977] Lustre: Skipped 2 previous similar messages [10678.179916] Lustre: server umount lustre-OST0000 complete [10682.874580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10682.878832] LustreError: Skipped 1 previous similar message [10682.882486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10682.886120] Lustre: Skipped 1 previous similar message [10686.434475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.457500] Lustre: Skipped 1 previous similar message [10688.240644] Lustre: server umount lustre-MDT0000 complete [10691.553517] LustreError: 285955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10691.589547] LustreError: 285955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10692.762756] LustreError: 285072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786336991 with bad export cookie 15506960752624613296 [10692.767443] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10692.770840] LustreError: 285072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10693.029599] Lustre: server umount lustre-MDT0001 complete [10713.739363] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10725.318973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.628438] LustreError: 289981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.667888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10725.673471] Lustre: Skipped 2 previous similar messages [10729.586679] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10738.907719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10743.974378] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10746.690422] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10753.724728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10760.744366] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10770.096081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10770.455280] Lustre: lustre-OST0001: new disk, initializing [10770.474843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10770.496699] Lustre: 292289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10771.948904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10771.964386] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10772.293914] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10779.690179] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10791.070711] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10794.820351] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10803.574575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10804.207652] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10813.458588] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10813.504139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10813.525293] Lustre: Skipped 4 previous similar messages [10817.901553] Lustre: server umount lustre-MDT0000 complete [10818.534327] LustreError: 289982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.543297] LustreError: 289982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10821.482639] LustreError: 290719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786337119 with bad export cookie 15506960752624614178 [10821.485916] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10821.489936] LustreError: 290719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10821.703793] Lustre: server umount lustre-MDT0001 complete [10834.790987] Lustre: server umount lustre-OST0000 complete [10847.832297] Lustre: server umount lustre-OST0001 complete [10864.171633] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [10876.086898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10876.342789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10876.353168] Lustre: Skipped 3 previous similar messages [10880.391402] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10885.703388] LustreError: 295347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10885.767134] LustreError: 295347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10888.437134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10892.666652] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10895.612672] Lustre: 296488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10901.725731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10908.385130] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10917.915807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10921.275898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10921.293243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10924.217399] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [10932.483252] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10936.494586] Lustre: 298364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10943.427686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10943.495264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10954.887339] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:47:32 (1786337252) [10970.594540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10970.599325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10970.607609] Lustre: Skipped 3 previous similar messages [10970.611144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10983.402570] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10983.540986] Lustre: server umount lustre-MDT0000 complete [10987.576937] LustreError: 295328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786337285 with bad export cookie 15506960752624620968 [10987.588041] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10987.591535] LustreError: 295328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10994.270687] Lustre: server umount lustre-MDT0001 complete [11005.040229] Lustre: server umount lustre-OST0000 complete [11008.711362] Lustre: server umount lustre-OST0001 complete [11015.285945] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [11024.828548] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [11071.431788] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [11080.620394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.965601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11080.985313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11081.042879] Lustre: lustre-MDT0000: new disk, initializing [11081.109435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11085.334683] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11096.405260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11096.514957] Lustre: 302603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11096.705860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11100.667811] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11111.923295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11112.198649] Lustre: 304236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11114.719575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11118.441172] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11127.741840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11127.910756] Lustre: 305108:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11129.909169] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11129.915710] Lustre: Skipped 1 previous similar message [11130.054061] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11133.342438] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11142.002663] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11145.727675] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11145.730836] Lustre: Skipped 1 previous similar message [11158.745745] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:50:56 (1786337456) [11161.597897] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11161.606322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11161.621216] Lustre: Skipped 7 previous similar messages [11161.649221] Lustre: Skipped 21 previous similar messages [11167.191710] Lustre: server umount lustre-MDT0000 complete [11170.482637] LustreError: 304247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786337468 with bad export cookie 15506960752624628437 [11170.489233] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11170.493382] LustreError: 304247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11170.808992] Lustre: server umount lustre-MDT0001 complete [11185.587946] Lustre: server umount lustre-OST0000 complete [11187.679133] Lustre: 267805:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786337470/real 1786337470] req@ffff895e87c9b100 x1873109525594496/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786337486 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11188.791281] Lustre: server umount lustre-OST0001 complete [11197.641816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11198.004908] LustreError: 307356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11198.037358] LustreError: 307356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11198.153565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11198.161741] Lustre: Skipped 7 previous similar messages [11203.440206] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11213.307383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11217.989792] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11225.959810] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11231.196470] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11239.899939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11246.445622] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11254.957980] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11275.970779] loop1: detected capacity change from 0 to 204800000 [11276.434829] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11276.441686] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11276.448728] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11282.660648] loop1: detected capacity change from 0 to 204800000 [11290.206425] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11290.262842] LustreError: 307346:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs789: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11290.274774] LustreError: 307346:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs789 [11290.285215] LustreError: 307346:0:(mgs_handler.c:517:mgs_target_reg()) fs789-MDT0001: Error checking target index: rc = -13 [11290.309036] LustreError: fs789-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11290.318321] LustreError: 311074:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11290.342440] LustreError: 311074:0:(tgt_mount.c:2129:server_put_super()) no obd fs789-MDT0001 [11290.360571] LustreError: 311074:0:(tgt_mount.c:116:server_deregister_mount()) fs789-MDT0001 not registered [11290.418491] Lustre: server umount fs789-MDT0001 complete [11290.422126] LustreError: 311074:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11299.680357] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11299.740140] Lustre: Setting parameter fs789-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs789-MDT0001 [11299.775802] Lustre: srv-fs789-MDT0001: No data found on store. Initialize space. [11299.778846] Lustre: Skipped 4 previous similar messages [11299.832093] Lustre: fs789-MDT0001: new disk, initializing [11299.834705] Lustre: Skipped 3 previous similar messages [11299.986895] Lustre: srv-fs789-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11299.992711] Lustre: 311540:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs789-MDT0001-mdtlov: can't create new llog: rc = -115 [11299.997641] LustreError: 311540:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs789-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11304.530346] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11311.384304] Lustre: server umount fs789-MDT0001 complete [11324.409789] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11324.486417] loop1: detected capacity change from 0 to 204800000 [11329.378136] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11329.640214] Lustre: 313108:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs789-MDT0001-mdtlov: can't create new llog: rc = -115 [11329.654742] LustreError: 313108:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs789-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11333.435961] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11339.450743] Lustre: server umount fs789-MDT0001 complete [11349.987432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11349.999815] Lustre: Skipped 3 previous similar messages [11350.005761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11350.005768] Lustre: Skipped 4 previous similar messages [11354.686563] Lustre: server umount lustre-MDT0000 complete [11362.871344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11362.947705] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11367.628791] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11368.445095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11375.228269] loop1: detected capacity change from 0 to 204800000 [11381.683293] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11382.103144] Lustre: 315556:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs789-MDT0001-mdtlov: can't create new llog: rc = -115 [11382.112521] LustreError: 315556:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs789-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11386.993323] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11392.193732] Lustre: server umount fs789-MDT0001 complete [11414.509651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11414.517510] LustreError: Skipped 1 previous similar message [11428.319143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11428.456344] Lustre: server umount lustre-OST0000 complete [11446.239126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11446.349666] Lustre: server umount lustre-MDT0000 complete [11450.117462] LustreError: 307336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786337748 with bad export cookie 15506960752624631118 [11450.122897] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11450.126722] LustreError: 307336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11450.401327] Lustre: server umount lustre-MDT0001 complete [11461.055459] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [11463.160729] Key type lgssc unregistered [11463.373676] LNet: 317905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11463.378619] LNetError: 317905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11463.389226] LNet: Removed LNI 192.168.203.128@tcp [11464.394475] Key type .llcrypt unregistered [11464.397641] Key type ._llcrypt unregistered [11483.390746] Key type ._llcrypt registered [11483.392864] Key type .llcrypt registered [11483.476731] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [11485.963903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11485.973627] alg: No test for adler32 (adler32-zlib) [11487.069834] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [11487.351187] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [11489.071365] Key type lgssc registered [11490.196873] Lustre: Echo OBD driver; http://www.lustre.org/ [11498.025308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11498.035705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11499.492303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11503.872830] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11508.820354] LustreError: 319179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11508.891949] LustreError: 319179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11512.763828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11513.149182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11516.968600] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11524.646955] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11530.811989] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11537.355495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11537.629873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11538.671943] LustreError: 321007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11544.035773] LustreError: 321063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11544.351824] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11550.696391] LustreError: 321008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11550.745921] LustreError: 321008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11552.136231] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11554.756577] LustreError: 321009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11565.167175] LustreError: 321008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11565.206046] LustreError: 321008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11574.102440] loop1: detected capacity change from 0 to 204800000 [11574.375944] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11574.383247] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11574.392573] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11580.021706] loop1: detected capacity change from 0 to 204800000 [11581.418307] LustreError: 321063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11581.440910] LustreError: 321063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11585.910298] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11586.032767] Lustre: fs8689-OST0000: new disk, initializing [11586.035872] Lustre: srv-fs8689-OST0000: No data found on store. Initialize space. [11586.041324] Lustre: 322913:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11586.113942] Lustre: fs8689-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11592.872731] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11600.784540] Lustre: server umount fs8689-OST0000 complete [11613.647760] loop1: detected capacity change from 0 to 204800000 [11616.375525] LustreError: 321063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11616.422170] LustreError: 321063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11619.284807] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11619.629529] Lustre: fs8689-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11622.384169] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11627.494181] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11628.095870] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11638.009678] loop2: detected capacity change from 0 to 204800000 [11638.421142] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11638.445887] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11638.455450] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11643.485327] loop2: detected capacity change from 0 to 204800000 [11648.898324] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11649.022710] LustreError: fs8689-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11649.037838] LustreError: fs8689-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11649.054903] LustreError: 325954:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11649.065608] LustreError: 325954:0:(tgt_mount.c:2129:server_put_super()) no obd fs8689-OST0001 [11649.078883] LustreError: 325954:0:(tgt_mount.c:116:server_deregister_mount()) fs8689-OST0001 not registered [11649.124629] Lustre: server umount fs8689-OST0001 complete [11649.135402] LustreError: 325954:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11659.697583] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11659.790316] Lustre: fs8689-OST0001: new disk, initializing [11659.793688] Lustre: srv-fs8689-OST0001: No data found on store. Initialize space. [11659.799108] Lustre: 326506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11659.926323] Lustre: fs8689-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11666.255808] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11673.416521] Lustre: server umount fs8689-OST0000 complete [11682.925446] LustreError: 321009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11682.976295] LustreError: 321009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11683.744987] Lustre: server umount fs8689-OST0001 complete [11696.453493] loop1: detected capacity change from 0 to 204800000 [11702.882767] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11703.163407] Lustre: fs8689-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11708.988459] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11716.200126] loop2: detected capacity change from 0 to 204800000 [11722.647356] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11728.281171] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11738.192723] Lustre: server umount fs8689-OST0000 complete [11748.532965] Lustre: server umount fs8689-OST0001 complete [11764.711398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11764.777469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11764.785223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11765.743730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11765.749573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11768.309324] Lustre: server umount lustre-OST0000 complete [11774.434341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11774.442947] LustreError: Skipped 1 previous similar message [11774.447581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11774.464417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11774.472692] Lustre: Skipped 1 previous similar message [11778.874627] Lustre: server umount lustre-MDT0000 complete [11782.944939] LustreError: 319160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786338081 with bad export cookie 14219107152733569239 [11782.947837] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11782.952827] LustreError: 319160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11783.431752] Lustre: server umount lustre-MDT0001 complete [11796.343118] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [11799.451816] Key type lgssc unregistered [11799.659669] LNet: 332445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11799.664315] LNetError: 332445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11799.676231] LNet: Removed LNI 192.168.203.128@tcp [11800.948147] Key type .llcrypt unregistered [11800.950157] Key type ._llcrypt unregistered [11832.017294] Key type ._llcrypt registered [11832.018787] Key type .llcrypt registered [11832.088748] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [11846.331851] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [11847.212542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11847.413218] alg: No test for adler32 (adler32-zlib) [11848.692557] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [11848.851133] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [11850.494315] Key type lgssc registered [11851.829212] Lustre: Echo OBD driver; http://www.lustre.org/ [11883.474036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11883.502581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11884.696995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11884.716434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11884.790911] Lustre: lustre-MDT0000: new disk, initializing [11884.844550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11884.859988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11888.748144] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11899.048816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11899.170340] Lustre: 336303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11899.218681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11899.222636] Lustre: Skipped 1 previous similar message [11899.298946] Lustre: lustre-MDT0001: new disk, initializing [11899.403679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11899.465453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11899.481939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11903.240630] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11912.340883] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11917.839520] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11925.067461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11925.385221] Lustre: lustre-OST0000: new disk, initializing [11925.390200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11925.394868] Lustre: 338273:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11925.484717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11932.948629] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [11933.890525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11933.902854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11934.061846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11942.201650] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11944.680815] Lustre: Mounted lustre-client - version 2.17.56_55_g704561f [11945.626669] Lustre: Unmounted lustre-client [11959.776658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11959.776942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.781119] LustreError: Skipped 1 previous similar message [11959.782270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11959.792863] Lustre: Skipped 1 previous similar message [11960.304740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11960.313962] Lustre: Skipped 1 previous similar message [11963.929886] Lustre: server umount lustre-OST0000 complete [11970.529260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11970.531271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11970.540612] Lustre: Skipped 1 previous similar message [11970.550527] Lustre: Skipped 2 previous similar messages [11976.864984] Lustre: server umount lustre-MDT0000 complete [11980.768454] LustreError: 338013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.783794] LustreError: 338013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11981.052193] LustreError: 337099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786338279 with bad export cookie 12752440116247577115 [11981.054922] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11981.066497] LustreError: 337099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11981.426212] Lustre: server umount lustre-MDT0001 complete [11990.721698] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [11993.361785] Key type lgssc unregistered [12004.960341] LNet: 340437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12004.969087] LNetError: 340437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12076.904775] LNet: Removed LNI 192.168.203.128@tcp [12077.841263] Key type .llcrypt unregistered [12077.850506] Key type ._llcrypt unregistered [12096.327585] Key type ._llcrypt registered [12096.330904] Key type .llcrypt registered [12096.445517] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [12112.042379] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [12113.285377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12113.357232] alg: No test for adler32 (adler32-zlib) [12115.188641] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [12115.719928] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [12117.607196] Key type lgssc registered [12118.602602] Lustre: Echo OBD driver; http://www.lustre.org/ [12150.219549] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:07:28 (1786338448) [12158.201568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12158.238827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12159.453413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12159.504743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12159.577460] Lustre: lustre-MDT0000: new disk, initializing [12159.635953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12159.649720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12164.017341] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12176.117720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12176.183635] Lustre: 343657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12176.221499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12176.225691] Lustre: Skipped 1 previous similar message [12176.273639] Lustre: lustre-MDT0001: new disk, initializing [12176.348821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12176.400531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12176.451923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12181.156422] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12191.691799] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12197.988432] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12205.753851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12205.946743] Lustre: lustre-OST0000: new disk, initializing [12205.950356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12205.955903] Lustre: 345626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12206.040534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12213.036975] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12214.348335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12214.365391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12214.685309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12224.434831] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12245.476470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12245.481881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.499460] LustreError: Skipped 1 previous similar message [12245.514236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12245.529247] Lustre: Skipped 1 previous similar message [12248.384793] Lustre: server umount lustre-OST0000 complete [12253.664832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12253.673353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12253.705713] Lustre: Skipped 1 previous similar message [12253.741336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.713180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12255.716977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.732825] Lustre: Skipped 1 previous similar message [12258.702294] Lustre: server umount lustre-MDT0000 complete [12261.802435] LustreError: 343651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786338560 with bad export cookie 8013312422630990392 [12261.806840] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12261.811480] LustreError: 343651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12262.201383] Lustre: server umount lustre-MDT0001 complete [12273.758930] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12275.951405] Key type lgssc unregistered [12276.231711] LNet: 347539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12276.240880] LNetError: 347539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12277.296224] LNet: Removed LNI 192.168.203.128@tcp [12278.534758] Key type .llcrypt unregistered [12278.539667] Key type ._llcrypt unregistered [12300.704798] Key type ._llcrypt registered [12300.708157] Key type .llcrypt registered [12300.873262] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [12302.523208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12302.578880] alg: No test for adler32 (adler32-zlib) [12303.784851] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [12304.000492] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [12305.647188] Key type lgssc registered [12306.636662] Lustre: Echo OBD driver; http://www.lustre.org/ [12314.452222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12314.465470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12315.895248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12319.924758] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12326.368994] LustreError: 348816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12326.378348] LustreError: 348816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12328.702557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12328.985240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12333.953419] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12342.224864] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12348.603750] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12357.874895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12358.117354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12365.153748] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12373.809306] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12383.698770] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12386.458598] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12392.071621] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.28@tcp (stopping) [12394.994423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12394.998484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12395.027802] Lustre: Skipped 2 previous similar messages [12395.036636] Lustre: Skipped 1 previous similar message [12397.168083] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.28@tcp (stopping) [12397.174535] Lustre: Skipped 1 previous similar message [12397.766508] Lustre: server umount lustre-MDT0000 complete [12400.100968] LustreError: 348813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12400.130996] LustreError: 348813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12402.288123] LustreError: 348811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12402.528426] LustreError: 351736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786338700 with bad export cookie 14693264023382718297 [12402.540628] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12402.549239] LustreError: 351736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12405.218185] LustreError: 350250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12405.224057] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12405.257764] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12408.982533] Lustre: server umount lustre-MDT0001 complete [12412.113550] Lustre: server umount lustre-OST0000 complete [12419.203242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12423.447501] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12426.003490] Lustre: 353003:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.128@tcp [12427.865124] Lustre: server umount MGS complete [12436.191986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12436.514540] LustreError: 353576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12436.535478] LustreError: 353576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12436.603509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12440.118844] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12446.182479] LustreError: 353577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12446.218261] LustreError: 353577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12448.378822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12448.638478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12453.122946] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12461.741201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12462.093235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12468.592886] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12477.046300] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12478.619375] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12484.924207] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12486.847637] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12491.240674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12491.248528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12491.252122] Lustre: Skipped 2 previous similar messages [12491.255196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12493.608826] Lustre: server umount lustre-OST0000 complete [12499.943158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12499.959163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12499.986806] Lustre: Skipped 1 previous similar message [12500.005799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12500.010984] Lustre: Skipped 1 previous similar message [12504.771825] Lustre: server umount lustre-MDT0000 complete [12507.111242] LustreError: 353576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12507.134918] LustreError: 353576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12508.389793] LustreError: 353558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786338806 with bad export cookie 14693264023382719137 [12508.482675] LustreError: 353558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12508.496239] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12509.036308] Lustre: server umount lustre-MDT0001 complete [12519.016746] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12521.729899] Key type lgssc unregistered [12522.168274] LNet: 357013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12522.185653] LNetError: 357013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12523.254318] LNet: Removed LNI 192.168.203.128@tcp [12523.931146] Key type .llcrypt unregistered [12523.933178] Key type ._llcrypt unregistered [12542.941788] Key type ._llcrypt registered [12542.943830] Key type .llcrypt registered [12543.107686] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [12560.610539] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [12562.393399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12562.407313] alg: No test for adler32 (adler32-zlib) [12563.476260] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [12563.751777] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [12565.528382] Key type lgssc registered [12566.729535] Lustre: Echo OBD driver; http://www.lustre.org/ [12602.266118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12602.288681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12603.537840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12603.595705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12603.675285] Lustre: lustre-MDT0000: new disk, initializing [12603.886868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12603.907737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12608.789128] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12619.332692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12619.431526] Lustre: 360180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12619.476816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12619.479792] Lustre: Skipped 1 previous similar message [12619.532931] Lustre: lustre-MDT0001: new disk, initializing [12619.610637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12619.635858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12619.644653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12624.112863] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12633.792973] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12639.582230] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12646.488110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12646.746364] Lustre: lustre-OST0000: new disk, initializing [12646.749653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12646.754372] Lustre: 362149:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12646.856819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12653.414347] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12656.164401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12656.173350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12656.232501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12662.568450] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12667.769222] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12667.984660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12672.290943] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12672.484981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12676.586126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12676.586250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12676.604174] LustreError: Skipped 1 previous similar message [12676.605831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12676.628242] Lustre: Skipped 1 previous similar message [12681.698870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12681.796590] Lustre: Skipped 2 previous similar messages [12686.844099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12688.351223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12688.516714] Lustre: server umount lustre-OST0000 complete [12696.545892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12696.553688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12696.572916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12696.584405] Lustre: Skipped 1 previous similar message [12702.177154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12702.184360] Lustre: Skipped 3 previous similar messages [12707.298064] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12707.402961] Lustre: server umount lustre-MDT0000 complete [12711.509378] LustreError: 360172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786339009 with bad export cookie 3860997801442330973 [12711.520523] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12711.537618] LustreError: 360172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12712.097862] Lustre: server umount lustre-MDT0001 complete [12726.609697] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:17:03 (1786339023) [12745.327777] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [12756.048729] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [12790.788067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12791.009583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12791.029627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12791.080178] Lustre: lustre-MDT0000: new disk, initializing [12791.131132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12791.143504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12795.350719] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12806.489388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12806.599559] Lustre: 366912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12806.661268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12806.664125] Lustre: Skipped 1 previous similar message [12806.713653] Lustre: lustre-MDT0001: new disk, initializing [12806.875469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12807.087836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12807.117624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12811.515258] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12820.228252] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12825.029388] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12832.466553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12832.705539] Lustre: lustre-OST0000: new disk, initializing [12832.709339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12832.714990] Lustre: 368878:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12832.777110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12838.082891] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [12838.961220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12838.967201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12839.046633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12846.357366] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12875.237280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12875.241515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12875.247339] Lustre: Skipped 1 previous similar message [12875.248838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12875.248843] Lustre: Skipped 2 previous similar messages [12877.060361] Lustre: server umount lustre-OST0000 complete [12883.936760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12883.942458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12883.949244] Lustre: Skipped 1 previous similar message [12886.800497] Lustre: server umount lustre-MDT0000 complete [12890.755862] LustreError: 369749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786339189 with bad export cookie 3860997801442331848 [12890.783466] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12891.016183] Lustre: server umount lustre-MDT0001 complete [12900.979174] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12903.199400] Key type lgssc unregistered [12903.474959] LNet: 371036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12903.482206] LNetError: 371036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12904.548773] LNet: Removed LNI 192.168.203.128@tcp [12905.346915] Key type .llcrypt unregistered [12905.357124] Key type ._llcrypt unregistered [12991.488468] Key type ._llcrypt registered [12991.490581] Key type .llcrypt registered [12991.717916] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [12992.147329] Key type .llcrypt unregistered [12992.151656] Key type ._llcrypt unregistered [13062.937854] Key type ._llcrypt registered [13062.944417] Key type .llcrypt registered [13063.080985] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13063.383548] Key type .llcrypt unregistered [13063.389979] Key type ._llcrypt unregistered [13091.362718] Key type ._llcrypt registered [13091.365099] Key type .llcrypt registered [13091.485710] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13091.759701] Key type .llcrypt unregistered [13091.796562] Key type ._llcrypt unregistered [13119.617709] Key type ._llcrypt registered [13119.620141] Key type .llcrypt registered [13119.693418] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13119.978751] Key type .llcrypt unregistered [13119.984791] Key type ._llcrypt unregistered [13193.163041] Key type ._llcrypt registered [13193.169640] Key type .llcrypt registered [13193.282631] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13193.608166] Key type .llcrypt unregistered [13193.612930] Key type ._llcrypt unregistered [13221.788825] Key type ._llcrypt registered [13221.790700] Key type .llcrypt registered [13221.925359] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13222.155520] Key type .llcrypt unregistered [13222.157740] Key type ._llcrypt unregistered [13250.812939] Key type ._llcrypt registered [13250.816053] Key type .llcrypt registered [13250.904521] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13251.170155] Key type .llcrypt unregistered [13251.172323] Key type ._llcrypt unregistered [13279.476417] Key type ._llcrypt registered [13279.478625] Key type .llcrypt registered [13279.561840] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13279.975382] Key type .llcrypt unregistered [13279.977341] Key type ._llcrypt unregistered [13307.848312] Key type ._llcrypt registered [13307.851647] Key type .llcrypt registered [13307.962193] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13308.175666] Key type .llcrypt unregistered [13308.177854] Key type ._llcrypt unregistered [13337.530938] Key type ._llcrypt registered [13337.533422] Key type .llcrypt registered [13337.627830] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13337.917523] Key type .llcrypt unregistered [13337.920610] Key type ._llcrypt unregistered [13398.409271] Key type ._llcrypt registered [13398.411645] Key type .llcrypt registered [13398.524472] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13398.798635] Key type .llcrypt unregistered [13398.801758] Key type ._llcrypt unregistered [13430.391286] Key type ._llcrypt registered [13430.395545] Key type .llcrypt registered [13430.487704] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13430.717406] Key type .llcrypt unregistered [13430.719272] Key type ._llcrypt unregistered [13459.276318] Key type ._llcrypt registered [13459.278647] Key type .llcrypt registered [13459.366246] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13459.643309] Key type .llcrypt unregistered [13459.646673] Key type ._llcrypt unregistered [13518.392753] Key type ._llcrypt registered [13518.395847] Key type .llcrypt registered [13518.520586] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [13533.100475] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [13534.467559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13534.615074] alg: No test for adler32 (adler32-zlib) [13535.710599] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [13535.985147] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [13537.631161] Key type lgssc registered [13538.781740] Lustre: Echo OBD driver; http://www.lustre.org/ [13571.501796] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13571.528502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13572.721173] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13572.744461] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13572.791563] Lustre: tst250fs-MDT0000: new disk, initializing [13572.856101] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13572.871309] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13577.632436] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [13587.282413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13587.466625] Lustre: 391243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13587.516445] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13587.520126] Lustre: Skipped 1 previous similar message [13587.572860] Lustre: tst250fs-MDT0001: new disk, initializing [13587.645577] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13587.672140] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13587.685701] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13591.574135] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [13596.044496] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13603.647818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13604.003937] Lustre: tst250fs-OST0000: new disk, initializing [13604.012194] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13604.022819] Lustre: 392875:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13604.192475] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13604.595030] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13604.637119] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13604.755641] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13610.333315] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [13621.606566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13621.828915] Lustre: tst250fs-OST0001: new disk, initializing [13621.831865] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13621.837057] Lustre: 393743:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13621.950859] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13627.770944] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [13630.527902] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13630.544931] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13630.673956] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13632.547922] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13635.095828] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13635.101324] Lustre: Skipped 2 previous similar messages [13655.010649] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13655.021666] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13655.034977] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13656.547663] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13656.559594] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13656.598024] Lustre: Skipped 2 previous similar messages [13656.618977] Lustre: Skipped 3 previous similar messages [13660.830266] Lustre: server umount tst250fs-MDT0000 complete [13664.187048] LustreError: 391234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786339962 with bad export cookie 4583486861302523450 [13664.187462] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13664.197730] LustreError: 391234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13664.518648] Lustre: server umount tst250fs-MDT0001 complete [13676.859876] Lustre: server umount tst250fs-OST0000 complete [13691.196504] Lustre: server umount tst250fs-OST0001 complete [13710.379194] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [13717.506407] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [13746.184929] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:34:04 (1786340044) [13747.818818] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13763.130726] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [13772.313420] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [13801.183668] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13506 sec ========= 01:34:59 (1786340099) [13803.385759] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:35:01 (1786340101) === [13806.878146] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:35:04 (1786340104) === [13842.813868] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [13845.268393] Key type lgssc unregistered [13845.570627] LNet: 402874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13845.610142] LNetError: 402874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13845.630919] LNet: Removed LNI 192.168.203.128@tcp [13846.443380] Key type .llcrypt unregistered [13846.446272] Key type ._llcrypt unregistered