[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501176641 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003438] x2apic enabled [ 0.005005] Switched APIC routing to physical x2apic. [ 0.007013] kvm-guest: setup PV IPIs [ 0.009827] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011018] pid_max: default: 32768 minimum: 301 [ 0.012157] LSM: Security Framework initializing [ 0.013062] Yama: becoming mindful. [ 0.014060] SELinux: Initializing. [ 0.015073] *** VALIDATE selinux *** [ 0.024168] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029145] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030221] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032126] *** VALIDATE tmpfs *** [ 0.033489] *** VALIDATE proc *** [ 0.034282] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.036282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039038] Spectre V2 : User space: Vulnerable [ 0.040013] Speculative Store Bypass: Vulnerable [ 0.043381] debug: unmapping init [mem 0xffffffff84a59000-0xffffffff84a60fff] [ 0.045220] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046766] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047047] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054403] rcu: Hierarchical SRCU implementation. [ 0.056630] smp: Bringing up secondary CPUs ... [ 0.057712] x86: Booting SMP configuration: [ 0.058040] .... node #0, CPUs: #1 #2 #3 [ 0.061551] smp: Brought up 1 node, 4 CPUs [ 0.063016] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.239022] node 0 deferred pages initialised in 172ms [ 0.243133] devtmpfs: initialized [ 0.244177] x86/mm: Memory block size: 128MB [ 0.246662] gcov: version magic: 0x41383552 [ 0.248278] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.250260] pinctrl core: initialized pinctrl subsystem [ 0.251258] [ 0.251936] ************************************************************* [ 0.252018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253017] ** ** [ 0.254019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255017] ** ** [ 0.256014] ** This means that this kernel is built to expose internal ** [ 0.257017] ** IOMMU data structures, which may compromise security on ** [ 0.258011] ** your system. ** [ 0.259014] ** ** [ 0.260014] ** If you see this message and you are not debugging the ** [ 0.261016] ** kernel, report this immediately to your vendor! ** [ 0.262014] ** ** [ 0.263015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264016] ************************************************************* [ 0.265735] NET: Registered protocol family 16 [ 0.266519] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.267063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.269458] cpuidle: using governor menu [ 0.270951] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.274542] PCI: Using configuration type 1 for base access [ 0.277147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.287195] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.290037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.294052] cryptd: max_cpu_qlen set to 1000 [ 0.295258] ACPI: Added _OSI(Module Device) [ 0.296022] ACPI: Added _OSI(Processor Device) [ 0.297016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.298013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.302176] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.304702] ACPI: Interpreter enabled [ 0.305067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.306013] ACPI: Using IOAPIC for interrupt routing [ 0.307096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.308269] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.318157] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.319046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.320025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.321080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.323474] acpiphp: Slot [2] registered [ 0.324165] acpiphp: Slot [5] registered [ 0.325174] acpiphp: Slot [6] registered [ 0.326197] acpiphp: Slot [7] registered [ 0.327163] acpiphp: Slot [8] registered [ 0.328179] acpiphp: Slot [9] registered [ 0.329218] acpiphp: Slot [10] registered [ 0.330145] acpiphp: Slot [3] registered [ 0.331130] acpiphp: Slot [4] registered [ 0.332153] acpiphp: Slot [11] registered [ 0.333115] acpiphp: Slot [12] registered [ 0.334115] acpiphp: Slot [13] registered [ 0.335116] acpiphp: Slot [14] registered [ 0.336127] acpiphp: Slot [15] registered [ 0.337113] acpiphp: Slot [16] registered [ 0.338134] acpiphp: Slot [17] registered [ 0.339160] acpiphp: Slot [18] registered [ 0.340105] acpiphp: Slot [19] registered [ 0.341132] acpiphp: Slot [20] registered [ 0.342120] acpiphp: Slot [21] registered [ 0.343127] acpiphp: Slot [22] registered [ 0.344113] acpiphp: Slot [23] registered [ 0.345126] acpiphp: Slot [24] registered [ 0.346106] acpiphp: Slot [25] registered [ 0.347165] acpiphp: Slot [26] registered [ 0.348103] acpiphp: Slot [27] registered [ 0.349116] acpiphp: Slot [28] registered [ 0.350142] acpiphp: Slot [29] registered [ 0.351099] acpiphp: Slot [30] registered [ 0.352139] acpiphp: Slot [31] registered [ 0.353086] PCI host bridge to bus 0000:00 [ 0.354020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.356035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.357032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.358036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.359032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.360181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.362026] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364285] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370637] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373684] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.374023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377996] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378810] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380739] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.394025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.399020] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.412025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.427028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.437526] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.451025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.467025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.475068] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.484019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.499018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.521016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.541772] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.550022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.556024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.571023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.580066] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.590017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.604018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.628018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.634955] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.641024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.647021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.658024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.667236] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.668341] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.670329] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.674478] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.676267] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.681066] iommu: Default domain type: Passthrough [ 0.682514] SCSI subsystem initialized [ 0.684178] ACPI: bus type USB registered [ 0.686132] usbcore: registered new interface driver usbfs [ 0.688138] usbcore: registered new interface driver hub [ 0.690082] usbcore: registered new device driver usb [ 0.692213] pps_core: LinuxPPS API ver. 1 registered [ 0.693020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.696126] PTP clock support registered [ 0.698190] EDAC MC: Ver: 3.0.0 [ 0.699417] PCI: Using ACPI for IRQ routing [ 0.700000] NetLabel: Initializing [ 0.700024] NetLabel: domain hash size = 128 [ 0.700039] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.702209] NetLabel: unlabeled traffic allowed by default [ 0.704359] vgaarb: loaded [ 0.705190] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.708025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.713603] clocksource: Switched to clocksource kvm-clock [ 0.826689] VFS: Disk quotas dquot_6.6.0 [ 0.827945] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.829795] *** VALIDATE ramfs *** [ 0.830713] *** VALIDATE hugetlbfs *** [ 0.831923] pnp: PnP ACPI init [ 0.834682] pnp: PnP ACPI: found 6 devices [ 0.849405] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.852225] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.853968] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.855628] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.857561] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.859393] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.861700] NET: Registered protocol family 2 [ 0.863811] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.868386] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.871307] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.876187] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.879857] TCP: Hash tables configured (established 65536 bind 65536) [ 0.882965] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.886216] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.889070] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.891580] NET: Registered protocol family 1 [ 0.894314] RPC: Registered named UNIX socket transport module. [ 0.896529] RPC: Registered udp transport module. [ 0.898264] RPC: Registered tcp transport module. [ 0.899725] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.902201] NET: Registered protocol family 44 [ 0.904061] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.906297] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.908505] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.911113] PCI: CLS 0 bytes, default 64 [ 0.913150] Unpacking initramfs... [ 2.437314] debug: unmapping init [mem 0xffff94f5fcc54000-0xffff94f5fffbffff] [ 2.441407] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.444012] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.447079] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.983696] Initialise system trusted keyrings [ 2.985371] Key type blacklist registered [ 2.987518] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.996290] zbud: loaded [ 2.999695] *** VALIDATE nfs *** [ 3.000979] *** VALIDATE nfs4 *** [ 3.002582] pstore: using deflate compression [ 3.006576] Platform Keyring initialized [ 3.127274] NET: Registered protocol family 38 [ 3.128943] Key type asymmetric registered [ 3.130586] Asymmetric key parser 'x509' registered [ 3.132109] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.135478] io scheduler mq-deadline registered [ 3.136941] io scheduler kyber registered [ 3.138641] io scheduler bfq registered [ 3.143679] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.146444] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.149079] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.151544] ACPI: Power Button [PWRF] [ 3.156756] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.164319] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.177197] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.184259] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.199165] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.228299] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.254772] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.259427] Non-volatile memory driver v1.3 [ 3.261124] Linux agpgart interface v0.103 [ 3.293257] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.295936] vda: detected capacity change from 0 to 74702848 [ 3.311568] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.314133] vdb: detected capacity change from 0 to 1073741824 [ 3.329436] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.332767] vdc: detected capacity change from 0 to 2621440000 [ 3.353200] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.355949] vdd: detected capacity change from 0 to 2621440000 [ 3.376422] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.379548] vde: detected capacity change from 0 to 4294967296 [ 3.398981] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.401661] vdf: detected capacity change from 0 to 4294967296 [ 3.407557] libphy: Fixed MDIO Bus: probed [ 3.413528] usbcore: registered new interface driver usbserial_generic [ 3.415977] usbserial: USB Serial support registered for generic [ 3.418194] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.422297] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.423832] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.426765] mousedev: PS/2 mouse device common for all mice [ 3.429862] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.431483] rtc_cmos 00:05: RTC can wake from S4 [ 3.436107] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.436287] rtc_cmos 00:05: registered as rtc0 [ 3.440445] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.442503] intel_pstate: CPU model not supported [ 3.444547] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.448604] hid: raw HID events driver (C) Jiri Kosina [ 3.450509] usbcore: registered new interface driver usbhid [ 3.452610] usbhid: USB HID core driver [ 3.453866] drop_monitor: Initializing network drop monitor service [ 3.456812] Initializing XFRM netlink socket [ 3.459210] NET: Registered protocol family 10 [ 3.462310] Segment Routing with IPv6 [ 3.464099] NET: Registered protocol family 17 [ 3.466381] mpls_gso: MPLS GSO support [ 3.472940] RAS: Correctable Errors collector initialized. [ 3.475234] AVX version of gcm_enc/dec engaged. [ 3.477091] AES CTR mode by8 optimization enabled [ 3.560429] sched_clock: Marking stable (3560400212, 0)->(4628310696, -1067910484) [ 3.565557] registered taskstats version 1 [ 3.567697] Loading compiled-in X.509 certificates [ 3.570159] zswap: loaded using pool lzo/zbud [ 3.599947] Key type big_key registered [ 3.618720] Key type encrypted registered [ 3.620570] ima: No TPM chip found, activating TPM-bypass! [ 3.622664] ima: Allocated hash algorithm: sha1 [ 3.624554] ima: No architecture policies found [ 3.626620] evm: Initialising EVM extended attributes: [ 3.628860] evm: security.selinux [ 3.630188] evm: security.ima [ 3.631354] evm: security.capability [ 3.632910] evm: HMAC attrs: 0x1 [ 3.635776] rtc_cmos 00:05: setting system clock to 2026-08-17 14:12:56 UTC (1786975976) [ 3.642577] debug: unmapping init [mem 0xffffffff85a03000-0xffffffff85bfffff] [ 3.646164] debug: unmapping init [mem 0xffffffff84782000-0xffffffff84a58fff] [ 3.655281] Write protecting the kernel read-only data: 28672k [ 3.658890] debug: unmapping init [mem 0xffffffff82e03000-0xffffffff82ffffff] [ 3.662106] debug: unmapping init [mem 0xffffffff83714000-0xffffffff837fffff] [ 3.695633] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.701458] systemd[1]: Detected virtualization kvm. [ 3.703840] systemd[1]: Detected architecture x86-64. [ 3.705967] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.728939] systemd[1]: No hostname configured. [ 3.730711] systemd[1]: Set hostname to . [ 3.732806] random: systemd: uninitialized urandom read (16 bytes read) [ 3.735318] systemd[1]: Initializing machine ID from random generator. [ 3.784537] random: ln: uninitialized urandom read (6 bytes read) [ 3.884820] random: systemd: uninitialized urandom read (16 bytes read) [ 3.887230] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.891320] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.894097] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.162981] device-mapper: uevent: version 1.0.3 [ 5.165590] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.429548] virtio_net virtio0 ens2: renamed from eth0 [ 7.660997] random: fast init done [ 8.145692] scsi host0: ata_piix [ 8.265217] scsi host1: ata_piix [ 8.271932] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.281101] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.167139] random: crng init done [ 13.174997] random: 7 urandom warning(s) missed due to ratelimiting [ 15.904572] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.403272] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.191335] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.254973] SELinux: Disabled at runtime. [ 22.448679] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.475057] systemd[1]: Detected virtualization kvm. [ 22.476943] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.242849] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.248747] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.275792] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.294586] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.305393] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.342751] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.370526] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ 24.816210] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.651819] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.181258] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.405943] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.202984] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.328617] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.788239] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 33.575732] NFS: Registering the id_resolver key type [ 33.578521] Key type id_resolver registered [ 33.580911] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg236-server login: [ 52.710022] hrtimer: interrupt took 8034474 ns [ 92.560134] libcfs: loading out-of-tree module taints kernel. [ 92.644669] Key type ._llcrypt registered [ 92.666827] Key type .llcrypt registered [ 92.835152] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 112.553375] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 114.520150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.535157] alg: No test for adler32 (adler32-zlib) [ 115.846301] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 116.637780] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 118.392357] Key type lgssc registered [ 120.184078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 139.018464] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 183.815461] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 198.572427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.612433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.010177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 200.046811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 200.184875] Lustre: lustre-MDT0000: new disk, initializing [ 200.320074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.342905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.578488] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.305805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.413911] Lustre: 6531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 221.447757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 221.452890] Lustre: Skipped 1 previous similar message [ 221.554691] Lustre: lustre-MDT0001: new disk, initializing [ 221.631991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.682129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 221.710908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 226.415199] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.303832] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 241.542334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.797466] Lustre: lustre-OST0000: new disk, initializing [ 241.801332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.807456] Lustre: 8467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.895803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.688901] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.423387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 250.444508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 250.551867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 263.891462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.072410] Lustre: lustre-OST0001: new disk, initializing [ 264.078099] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.082812] Lustre: 9540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.166790] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.038879] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 272.449346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 272.459431] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 272.542223] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 283.744980] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 290.666369] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 297.110975] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.651557] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing yml_node [ 307.392676] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 310.043215] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 312.954802] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 314.837646] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 17 10:18:05 EDT 2026 [ 335.469728] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 337.645437] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 354.790668] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.790784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 354.794672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.803538] Lustre: Skipped 1 previous similar message [ 359.907260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.914509] Lustre: Skipped 6 previous similar messages [ 360.997988] Lustre: server umount lustre-MDT0000 complete [ 365.259354] LustreError: 6524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976338 with bad export cookie 11457055580600753539 [ 365.260276] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 365.268354] LustreError: 6524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 370.144555] LustreError: 6537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.144799] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.151042] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.166512] LustreError: 6537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 370.208214] Lustre: Skipped 3 previous similar messages [ 371.618794] Lustre: server umount lustre-MDT0001 complete [ 375.317334] Lustre: server umount lustre-OST0000 complete [ 378.761456] Lustre: server umount lustre-OST0001 complete [ 384.262516] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 393.640160] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 428.941354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.276410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 429.308755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 429.468416] Lustre: lustre-MDT0000: new disk, initializing [ 429.577895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.595870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 433.647763] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 446.026142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.151351] Lustre: 15098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 446.243203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 446.250772] Lustre: Skipped 1 previous similar message [ 446.393386] Lustre: lustre-MDT0001: new disk, initializing [ 446.484615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 446.538516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 446.573371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 451.037467] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 460.269592] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 466.292906] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 473.233947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 473.577207] Lustre: lustre-OST0000: new disk, initializing [ 473.581822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 473.590547] Lustre: 17071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 473.687600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.016893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 475.029913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 475.225112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 479.800591] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 489.190811] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 494.326474] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 494.493874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 499.212143] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 499.366169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 505.830500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.830669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.836354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 505.836363] Lustre: Skipped 1 previous similar message [ 505.836507] LustreError: Skipped 1 previous similar message [ 505.862815] Lustre: Skipped 1 previous similar message [ 510.959117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.968626] Lustre: Skipped 2 previous similar messages [ 516.077668] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 516.267062] Lustre: server umount lustre-OST0000 complete [ 521.194562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.212418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 521.225646] Lustre: Skipped 1 previous similar message [ 521.228711] Lustre: Skipped 2 previous similar messages [ 526.310607] LustreError: 15110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.326755] LustreError: 15110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 526.395478] Lustre: server umount lustre-MDT0000 complete [ 530.197841] LustreError: 15090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976503 with bad export cookie 11457055580600755079 [ 530.202269] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.210562] LustreError: 15090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 530.551550] Lustre: server umount lustre-MDT0001 complete [ 536.964808] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:21:47 (1786976507) [ 553.512464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.814891] LustreError: 19945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.864569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 557.577669] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 559.094203] LustreError: 19946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.173073] LustreError: 19945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.737978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.725694] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 578.708454] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 585.764897] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 588.676157] Lustre: Failing over lustre-MDT0000 [ 588.697064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.714081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.719367] Lustre: Skipped 2 previous similar messages [ 589.049572] Lustre: server umount lustre-MDT0000 complete [ 590.757969] LustreError: 19946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.798455] LustreError: 19946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 606.185150] Lustre: 3662:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786976563/real 1786976563] req@ffff94f5437bc700 x1873780239661312/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786976579 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 606.222634] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.453154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.340056] LustreError: 20672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.366367] LustreError: 20672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 617.441106] LustreError: 3661:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94f67bb0ea00 x1873780239664000/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 617.857237] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 623.081975] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 623.104995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 623.156346] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 623.400167] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 636.270718] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 640.485307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.491657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.512566] Lustre: Skipped 2 previous similar messages [ 647.264956] Lustre: server umount lustre-MDT0000 complete [ 650.722183] LustreError: 20672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.753173] LustreError: 20672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 651.751249] LustreError: 19925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976624 with bad export cookie 11457055580600756493 [ 651.773389] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 651.780756] LustreError: 19925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 652.383410] Lustre: server umount lustre-MDT0001 complete [ 662.302109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.101752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 663.108140] Lustre: Skipped 2 previous similar messages [ 668.764417] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 676.864477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.227894] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 687.900708] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 693.747909] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 696.179839] Lustre: Failing over lustre-MDT0000 [ 696.487861] Lustre: server umount lustre-MDT0000 complete [ 697.825520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.184179] Lustre: 3665:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786976670/real 1786976670] req@ffff94f67faeb480 x1873780239690624/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786976686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 713.224255] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 715.097229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 718.305497] LustreError: 23283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.332594] LustreError: 23283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 723.435931] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9effa359359d6cb7 [ 723.446420] Lustre: MGC192.168.202.136@tcp: Connection restored to 0@lo (at 0@lo) [ 723.453739] Lustre: Skipped 1 previous similar message [ 723.701895] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 728.616517] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 729.063085] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 729.073400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 729.128098] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 738.913536] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 744.420025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 744.424312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 744.433627] Lustre: Skipped 1 previous similar message [ 744.439203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 744.447667] Lustre: Skipped 3 previous similar messages [ 749.748581] Lustre: server umount lustre-MDT0000 complete [ 754.595589] LustreError: 23263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976727 with bad export cookie 11457055580600757431 [ 754.633527] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 754.638835] LustreError: 23263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 755.098868] Lustre: server umount lustre-MDT0001 complete [ 768.833372] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:25:38 (1786976738) [ 787.758610] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 797.575936] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 841.357719] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 852.426712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.612549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 852.648120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 852.722232] Lustre: lustre-MDT0000: new disk, initializing [ 852.804190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.806996] Lustre: Skipped 2 previous similar messages [ 852.823449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 856.751792] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 868.569810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.774233] Lustre: 29665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 869.170543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 873.519738] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 884.516706] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 889.715403] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 895.850358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 896.095741] Lustre: 31633:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 897.859583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 897.900425] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 897.938989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 897.948338] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 902.114432] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 911.565842] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 916.862310] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 938.982663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.991639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 938.996769] Lustre: Skipped 1 previous similar message [ 939.017270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 939.026678] Lustre: Skipped 3 previous similar messages [ 941.540315] Lustre: server umount lustre-OST0000 complete [ 946.145589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 946.154623] LustreError: Skipped 1 previous similar message [ 951.181618] LustreError: 32862:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff94f67f4d0d00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 951.325233] Lustre: server umount lustre-MDT0000 complete [ 954.337340] LustreError: 29676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.363985] LustreError: 29676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 955.696426] LustreError: 29658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976928 with bad export cookie 11457055580600757837 [ 955.700319] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 955.709384] LustreError: 29658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 956.561187] Lustre: server umount lustre-MDT0001 complete [ 969.198412] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 972.445883] Key type lgssc unregistered [ 972.707483] LNet: 33618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.716804] LNetError: 33618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.728538] LNet: Removed LNI 192.168.202.136@tcp [ 973.590254] Key type .llcrypt unregistered [ 973.592686] Key type ._llcrypt unregistered [ 1001.040065] Key type ._llcrypt registered [ 1001.041586] Key type .llcrypt registered [ 1001.232143] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1002.759371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1002.942725] alg: No test for adler32 (adler32-zlib) [ 1004.170876] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 1004.496897] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1006.265881] Key type lgssc registered [ 1007.619776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1015.816603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1015.839791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.388437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.489061] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1027.556286] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.572884] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1031.355921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.834313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1036.197523] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1044.035538] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1050.149504] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1057.280517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1057.625392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.835885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1065.449492] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1074.818355] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1077.128043] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1090.387753] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:31:01 (1786977061) [ 1092.901111] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1094.728497] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1102.454964] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:31:12 (1786977072) [ 1113.070019] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:31:23 (1786977083) [ 1114.405883] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1124.059230] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1124.064596] Lustre: Skipped 2 previous similar messages [ 1125.947315] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:31:36 (1786977096) [ 1128.567983] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1128.571258] Lustre: Skipped 2 previous similar messages [ 1137.562938] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1137.569441] Lustre: Skipped 4 previous similar messages [ 1153.289587] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:32:03 (1786977123) [ 1156.257372] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1159.722700] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1168.503438] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:32:19 (1786977139) [ 1170.724824] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1170.738455] Lustre: Skipped 6 previous similar messages [ 1187.690267] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:32:38 (1786977158) [ 1205.218609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1205.226503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.241373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.251525] Lustre: Skipped 1 previous similar message [ 1209.915583] Lustre: server umount lustre-OST0000 complete [ 1215.459537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.466548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.472111] Lustre: Skipped 2 previous similar messages [ 1215.484151] Lustre: Skipped 1 previous similar message [ 1220.579315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1220.585071] Lustre: Skipped 1 previous similar message [ 1221.261662] Lustre: server umount lustre-MDT0000 complete [ 1225.442347] LustreError: 35013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977198 with bad export cookie 16129240324779736821 [ 1225.462261] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1225.870543] Lustre: server umount lustre-MDT0001 complete [ 1238.015655] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1241.263762] Key type lgssc unregistered [ 1241.659421] LNet: 42894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.677986] LNetError: 42894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.705438] LNet: Removed LNI 192.168.202.136@tcp [ 1242.712215] Key type .llcrypt unregistered [ 1242.716490] Key type ._llcrypt unregistered [ 1261.077487] Key type ._llcrypt registered [ 1261.079241] Key type .llcrypt registered [ 1261.168541] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 1278.658326] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1280.000244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.044029] alg: No test for adler32 (adler32-zlib) [ 1281.129962] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 1281.357327] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1283.027205] Key type lgssc registered [ 1284.440182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1320.964091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1321.015673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.392670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1322.439172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1322.602220] Lustre: lustre-MDT0000: new disk, initializing [ 1322.788065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.828787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1327.197253] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1338.164819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.284757] Lustre: 46049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1338.344892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1338.350444] Lustre: Skipped 1 previous similar message [ 1338.372147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1338.444755] Lustre: lustre-MDT0001: new disk, initializing [ 1338.543419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1338.595157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1338.615526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1342.890967] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1352.694760] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1358.085411] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1364.738971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1365.003446] Lustre: lustre-OST0000: new disk, initializing [ 1365.007650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1365.017484] Lustre: 48021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1365.121392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.378591] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1372.258479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1372.275136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1372.431700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1381.256542] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1385.717446] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1385.969972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1390.600638] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1390.861878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1393.127777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1393.138362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.168522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1396.193692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1396.211822] Lustre: Skipped 2 previous similar messages [ 1399.032696] Lustre: server umount lustre-OST0000 complete [ 1405.424089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1405.433429] LustreError: Skipped 1 previous similar message [ 1405.438142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.454569] Lustre: Skipped 1 previous similar message [ 1405.471244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1409.614588] Lustre: server umount lustre-MDT0000 complete [ 1413.663743] LustreError: 46040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977386 with bad export cookie 6102584422277153431 [ 1413.672342] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1413.673140] LustreError: 46040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1414.129793] Lustre: server umount lustre-MDT0001 complete [ 1427.041865] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:36:36 (1786977396) [ 1435.637630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.342354] LustreError: 50482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.359178] LustreError: 50482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1436.424338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.774420] LustreError: 50483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.776400] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1446.884738] LustreError: 50482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.008053] LustreError: 50483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.907843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.361395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1458.435943] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1466.619437] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1472.839908] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1479.805725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1480.324322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.849362] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1495.399145] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1503.515925] LustreError: 53011:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1504.593162] Lustre: Modifying parameter general.timeout=1 in log params [ 1505.600367] Lustre: Modifying parameter general.timeout=2 in log params [ 1507.881051] Lustre: Modifying parameter general.timeout=4 in log params [ 1507.887376] Lustre: Skipped 1 previous similar message [ 1512.407787] Lustre: Modifying parameter general.timeout=8 in log params [ 1512.413099] Lustre: Skipped 3 previous similar messages [ 1520.614352] Lustre: Modifying parameter general.timeout=15 in log params [ 1520.625238] Lustre: Skipped 6 previous similar messages [ 1537.718430] Lustre: Modifying parameter general.timeout=31 in log params [ 1537.729014] Lustre: Skipped 15 previous similar messages [ 1569.912424] Lustre: Modifying parameter general.timeout=65 in log params [ 1569.915962] Lustre: Skipped 33 previous similar messages [ 1634.717950] Lustre: Modifying parameter general.timeout=128 in log params [ 1634.728728] Lustre: Skipped 62 previous similar messages [ 1648.498153] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:40:18 (1786977618) [ 1651.475845] Lustre: 59966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1653.526585] Lustre: 60064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1655.601488] Lustre: 60162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1661.460422] Lustre: 60477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1661.471151] Lustre: 60477:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1689.436405] Lustre: 61840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1689.501462] LustreError: 61840:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1693.942948] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:41:04 (1786977664) [ 1709.550639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1709.561504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.579626] Lustre: Skipped 1 previous similar message [ 1709.586792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.597928] Lustre: Skipped 2 previous similar messages [ 1713.150837] Lustre: server umount lustre-MDT0000 complete [ 1716.201951] LustreError: 51478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.210824] LustreError: 51478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1717.212149] LustreError: 50464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977690 with bad export cookie 6102584422277154313 [ 1717.221547] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1717.227086] LustreError: 50464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1717.541324] Lustre: server umount lustre-MDT0001 complete [ 1731.537090] Lustre: server umount lustre-OST0000 complete [ 1763.174903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.630299] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1763.669442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1763.684923] Lustre: Skipped 24 previous similar messages [ 1763.909433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.101437] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1778.504437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1778.632987] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1778.680609] Lustre: 64372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1778.819787] LustreError: 64383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1784.249668] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1792.639264] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1798.937850] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1806.094624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.316669] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1806.602373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.605698] Lustre: Skipped 1 previous similar message [ 1812.805628] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1822.190614] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1824.305400] Lustre: 66942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1834.978502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1834.983786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.007220] Lustre: Skipped 2 previous similar messages [ 1835.013531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1835.030680] Lustre: Skipped 3 previous similar messages [ 1845.217081] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1845.415149] Lustre: server umount lustre-OST0000 complete [ 1850.854775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.869605] Lustre: Skipped 2 previous similar messages [ 1850.870495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1850.889548] LustreError: Skipped 1 previous similar message [ 1855.189541] Lustre: server umount lustre-MDT0000 complete [ 1855.969379] LustreError: 64383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1855.980334] LustreError: 64383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1858.662987] LustreError: 65130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977831 with bad export cookie 6102584422277157519 [ 1858.664508] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1858.673060] LustreError: 65130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1858.898751] Lustre: server umount lustre-MDT0001 complete [ 1869.394504] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1872.211572] Key type lgssc unregistered [ 1872.673909] LNet: 68153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.682963] LNetError: 68153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.698727] LNet: Removed LNI 192.168.202.136@tcp [ 1873.328229] Key type .llcrypt unregistered [ 1873.332195] Key type ._llcrypt unregistered [ 1899.326125] Key type ._llcrypt registered [ 1899.329632] Key type .llcrypt registered [ 1899.470738] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1900.715777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1900.755861] alg: No test for adler32 (adler32-zlib) [ 1901.927749] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 1902.170612] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1903.832306] Key type lgssc registered [ 1904.936260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1914.603761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1914.620187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.415065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.848823] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1926.626465] LustreError: 69568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.635676] LustreError: 69568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1931.771225] LustreError: 69569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.544032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.480286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1940.764337] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1949.250089] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1955.810602] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1962.965624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1963.587653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.724132] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1978.163352] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1997.091798] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2007.010071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2007.029889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.062424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.533933] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.559869] Lustre: Skipped 1 previous similar message [ 2007.570352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.579977] Lustre: Skipped 1 previous similar message [ 2012.645307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2013.208568] Lustre: server umount lustre-MDT0000 complete [ 2017.770287] LustreError: 70300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.802737] LustreError: 70300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2018.411481] LustreError: 69548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977991 with bad export cookie 7083392047351720717 [ 2018.428466] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2018.838406] Lustre: server umount lustre-MDT0001 complete [ 2033.966145] Lustre: server umount lustre-OST0000 complete [ 2061.845476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.091318] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2062.130405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2062.411416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.411685] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2075.729788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.814789] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2075.850500] Lustre: 74465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2076.251122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2081.119717] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2088.878711] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2094.332927] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2101.579954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2101.880260] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2102.236889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.418481] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2115.710937] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2117.493931] Lustre: 77038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2119.199186] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2123.746072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2123.752156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.761836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2123.767881] Lustre: Skipped 2 previous similar messages [ 2129.380558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2129.387918] Lustre: Skipped 3 previous similar messages [ 2138.083744] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2138.297892] Lustre: server umount lustre-OST0000 complete [ 2144.743336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.747369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.756039] Lustre: Skipped 2 previous similar messages [ 2144.780385] Lustre: Skipped 3 previous similar messages [ 2149.237900] Lustre: server umount lustre-MDT0000 complete [ 2149.858515] LustreError: 74472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.876331] LustreError: 74472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2153.079778] LustreError: 74458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978125 with bad export cookie 7083392047351721564 [ 2153.094150] LustreError: 74458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2153.099723] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.607763] Lustre: server umount lustre-MDT0001 complete [ 2167.190544] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2170.993611] Key type lgssc unregistered [ 2171.356545] LNet: 78253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.362585] LNetError: 78253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.380686] LNet: Removed LNI 192.168.202.136@tcp [ 2172.592362] Key type .llcrypt unregistered [ 2172.594989] Key type ._llcrypt unregistered [ 2200.884774] Key type ._llcrypt registered [ 2200.888439] Key type .llcrypt registered [ 2200.980379] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2201.906648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2202.080429] alg: No test for adler32 (adler32-zlib) [ 2203.106262] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 2203.354628] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2204.994418] Key type lgssc registered [ 2206.122765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2214.178994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2214.195615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.662202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.758678] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2226.150060] LustreError: 79673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.176198] LustreError: 79673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2229.848899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.396666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2235.507267] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2243.637434] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.953895] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2259.008509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2259.815619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.292994] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2277.307650] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2282.573799] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2283.405899] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2284.482911] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2287.288992] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2287.294420] Lustre: Skipped 2 previous similar messages [ 2291.630955] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2291.638575] Lustre: Skipped 3 previous similar messages [ 2300.126470] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2300.130597] Lustre: Skipped 8 previous similar messages [ 2316.460158] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2316.465981] Lustre: Skipped 17 previous similar messages [ 2348.634478] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2348.643432] Lustre: Skipped 32 previous similar messages [ 2387.459863] Lustre: 87777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2390.515762] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:52:41 (1786978361) [ 2392.247888] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2394.403401] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:52:44 (1786978364) [ 2400.226283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2400.233376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.255387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2402.273497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2402.278751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2402.294106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.892896] Lustre: server umount lustre-OST0000 complete [ 2409.442344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2409.446641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.454538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.461757] Lustre: Skipped 1 previous similar message [ 2412.527249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2412.527605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2417.639102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2417.644458] Lustre: Skipped 2 previous similar messages [ 2423.264233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2423.496989] Lustre: server umount lustre-MDT0000 complete [ 2427.332190] LustreError: 79653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978400 with bad export cookie 2898478024561648528 [ 2427.349957] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2427.716916] Lustre: server umount lustre-MDT0001 complete [ 2439.885331] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2443.145654] Key type lgssc unregistered [ 2443.517993] LNet: 89029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2443.524623] LNetError: 89029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.596729] LNet: Removed LNI 192.168.202.136@tcp [ 2445.886169] Key type .llcrypt unregistered [ 2445.888484] Key type ._llcrypt unregistered [ 2451.269764] Key type ._llcrypt registered [ 2451.273881] Key type .llcrypt registered [ 2451.413675] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2456.481769] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules [ 2457.579272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2457.657438] alg: No test for adler32 (adler32-zlib) [ 2458.693975] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 2458.911042] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2460.576363] Key type lgssc registered [ 2461.952322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2463.338775] LustreError: 90232:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2482.360382] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2485.749158] LustreError: 90232:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2491.499509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2491.514746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.017284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.033219] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2503.141541] LustreError: 91256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.168364] LustreError: 91256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2507.184886] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:54:37 (1786978477) [ 2508.259137] LustreError: 91255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.804456] LustreError: 91287:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2511.812693] LustreError: 91287:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2511.824740] LustreError: 91287:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2512.296170] Lustre: server umount lustre-MDT0000 complete [ 2530.467565] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2535.375172] Key type lgssc unregistered [ 2535.729730] LNet: 92787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2535.748987] LNetError: 92787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.768908] LNet: Removed LNI 192.168.202.136@tcp [ 2537.281476] Key type .llcrypt unregistered [ 2537.288197] Key type ._llcrypt unregistered [ 2563.411502] Key type ._llcrypt registered [ 2563.413133] Key type .llcrypt registered [ 2563.588323] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2564.893179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2565.097528] alg: No test for adler32 (adler32-zlib) [ 2566.241980] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 2566.625817] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2568.344264] Key type lgssc registered [ 2569.495271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2577.842904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2577.860804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.486497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.480992] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2589.669500] LustreError: 94007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.698887] LustreError: 94007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2594.788401] LustreError: 94008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.120157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.241672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2602.144274] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2610.079393] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2616.592505] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2623.686557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.503181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2630.983633] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2638.853123] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2692.210555] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:57:42 (1786978662) [ 2694.308546] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2696.563586] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:57:47 (1786978667) [ 2702.308562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.320628] Lustre: Skipped 2 previous similar messages [ 2702.320989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2702.337347] Lustre: Skipped 2 previous similar messages [ 2707.429571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2707.441647] Lustre: Skipped 1 previous similar message [ 2708.025270] Lustre: server umount lustre-MDT0000 complete [ 2712.472257] LustreError: 93988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978685 with bad export cookie 10010375900260923816 [ 2712.473957] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2712.483558] LustreError: 93988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2712.550720] LustreError: 94003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.560451] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.581792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2712.752177] Lustre: server umount lustre-MDT0001 complete [ 2719.745268] Lustre: server umount lustre-OST0000 complete [ 2734.028686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.784584] LustreError: 98304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.180221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2740.198342] LustreError: 98305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.219848] LustreError: 98305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2741.425295] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2749.420514] LustreError: 98304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.447617] LustreError: 98304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2753.008211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.982142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2758.652222] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2768.962636] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2776.090659] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2791.616540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2791.817302] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2791.828318] LustreError: 98292:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2791.851744] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2791.858668] LustreError: 100388:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2791.864716] LustreError: 100388:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2791.869620] LustreError: 100388:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2791.947770] Lustre: server umount lustre-OST0000 complete [ 2791.951207] LustreError: 100388:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2802.185968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.529583] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2802.535757] LustreError: 98294:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2802.569501] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2802.582800] LustreError: 100855:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2802.592522] LustreError: 100855:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2802.609238] LustreError: 100855:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2802.740349] Lustre: server umount lustre-OST0000 complete [ 2802.750263] LustreError: 100855:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2814.347745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2814.664824] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2814.669925] Lustre: Found index 0 for lustre-OST0000, updating log [ 2814.679429] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2814.833035] Lustre: lustre-OST0000: new disk, initializing [ 2814.837368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2814.848197] Lustre: 101390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2815.194310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.855275] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2827.164138] Lustre: Failing over lustre-OST0000 [ 2827.372685] Lustre: server umount lustre-OST0000 complete [ 2835.426123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2835.429095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.470989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.482088] Lustre: Skipped 1 previous similar message [ 2837.837646] Lustre: server umount lustre-MDT0000 complete [ 2840.554389] LustreError: 100710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2840.563268] LustreError: 100710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2841.895751] LustreError: 98284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978814 with bad export cookie 10010375900260926560 [ 2841.905727] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2841.916528] LustreError: 98284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2851.541844] Lustre: server umount lustre-MDT0001 complete [ 2864.982548] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:00:34 (1786978834) [ 2887.245203] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2901.405365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.719214] LustreError: 103752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2901.829722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.949446] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2916.290890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.116210] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2926.190598] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2933.827343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2934.662720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.670978] Lustre: Skipped 1 previous similar message [ 2934.692672] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2941.798894] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2941.837018] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2941.837145] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2942.390935] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2952.895062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2953.051108] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2953.157389] Lustre: lustre-OST0001: new disk, initializing [ 2953.160500] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2953.169978] Lustre: 106067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2961.256162] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2962.071868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2962.086213] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2962.220525] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2971.303698] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2982.313843] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2988.010423] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.020899] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2988.038912] Lustre: Skipped 2 previous similar messages [ 2988.059412] Lustre: Skipped 2 previous similar messages [ 2994.061595] Lustre: server umount lustre-MDT0001 complete [ 2995.350549] LustreError: 103748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.377497] LustreError: 103748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3004.691475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.867660] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3004.879716] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3004.962507] Lustre: 103739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3005.723401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3005.728245] Lustre: Skipped 1 previous similar message [ 3006.708026] LustreError: 107625:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3006.720220] LustreError: 107625:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3006.737333] LustreError: 103744:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3006.905348] LustreError: 107645:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.136@tcp: cfg command failed: rc = -17 [ 3006.918623] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3006.918623] [ 3011.068964] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3011.416686] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3024.952138] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:03:14 (1786978994) [ 3046.924208] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3067.712782] Lustre: 109686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3092.552486] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3097.553881] Lustre: 110825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3205.090325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3205.095542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.113711] Lustre: Skipped 1 previous similar message [ 3205.117620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.121473] Lustre: Skipped 4 previous similar messages [ 3210.495409] Lustre: server umount lustre-MDT0000 complete [ 3213.801276] LustreError: 106547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.810143] LustreError: 106547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3213.811751] LustreError: 93316:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff94f54505a300 x1873782808896896/t0(0) o38->lustre-MDT0000-lwp-MDT0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3213.815221] LustreError: 103733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979186 with bad export cookie 10010375900260927561 [ 3213.817348] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3213.823193] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.823202] Lustre: Skipped 3 previous similar messages [ 3213.875095] LustreError: 103733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3214.076265] Lustre: server umount lustre-MDT0001 complete [ 3228.382079] Lustre: server umount lustre-OST0000 complete [ 3232.864748] Lustre: server umount lustre-OST0001 complete [ 3238.829863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3250.545455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3261.192501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3275.088722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3302.149475] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3315.032800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.089454] Lustre: lustre-MDT0000: reset Object Index mappings [ 3315.883973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.385381] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3330.829063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.865309] Lustre: lustre-MDT0001: reset Object Index mappings [ 3337.220853] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3341.279347] Lustre: 115962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3350.762882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3357.175835] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3358.166679] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3360.261969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3368.806074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3375.119425] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3375.144598] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3378.012115] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3389.256126] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3393.570681] Lustre: 117840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3454.250548] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:10:24 (1786979424) [ 3480.033330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3480.040814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.051775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3480.057172] Lustre: Skipped 5 previous similar messages [ 3494.368620] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3494.632706] Lustre: server umount lustre-MDT0000 complete [ 3497.963717] LustreError: 115550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.980135] LustreError: 115550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3498.542700] LustreError: 117842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979471 with bad export cookie 10010375900260974426 [ 3498.544653] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3498.564473] LustreError: 117842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3505.222860] Lustre: server umount lustre-MDT0001 complete [ 3516.353592] Lustre: server umount lustre-OST0000 complete [ 3526.965646] Lustre: server umount lustre-OST0001 complete [ 3533.663242] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 3545.552345] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3590.984049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.259450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3591.308475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3591.404132] Lustre: lustre-MDT0000: new disk, initializing [ 3591.542820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.549982] Lustre: Skipped 3 previous similar messages [ 3591.579563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3596.583389] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3604.738847] Lustre: server umount lustre-MDT0000 complete [ 3615.603427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3616.152613] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3616.181566] Lustre: 122114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3616.211520] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3622.241619] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3628.427675] Lustre: server umount lustre-MDT0000 complete [ 3640.053297] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:13:30 (1786979610) [ 3642.190843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3644.364941] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:13:34 (1786979614) [ 3668.939648] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3683.788925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.417353] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3701.841102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3702.035629] Lustre: 124188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3702.117494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3702.121957] Lustre: Skipped 1 previous similar message [ 3702.224202] Lustre: lustre-MDT0001: new disk, initializing [ 3702.442977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3702.459140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3707.839984] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3713.561475] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3720.533696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3720.855638] Lustre: lustre-OST0000: new disk, initializing [ 3720.860238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3720.868890] Lustre: 125765:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3722.503337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3722.520624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3722.698683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3728.166233] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3740.056495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3740.228540] Lustre: lustre-OST0001: new disk, initializing [ 3740.231706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3740.237303] Lustre: 126634:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3741.780031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3741.794086] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3741.895518] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3747.438845] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3758.355991] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3762.923529] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4163.525174] Lustre: 133064:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4163.540418] Lustre: 133064:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4163.550549] Lustre: 133064:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4163.568800] Lustre: 133064:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4163.581438] Lustre: 133064:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4163.600347] Lustre: 133064:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4435.549842] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:26:45 (1786980405) [ 4453.860797] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.866757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4453.883739] Lustre: Skipped 8 previous similar messages [ 4453.904529] Lustre: Skipped 17 previous similar messages [ 4458.095957] Lustre: server umount lustre-MDT0000 complete [ 4458.985921] LustreError: 127442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4459.025927] LustreError: 127442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4462.178500] LustreError: 124179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786980435 with bad export cookie 10010375900261044286 [ 4462.183366] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4462.186961] LustreError: 124179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4464.116308] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4464.117702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4464.123936] Lustre: Skipped 1 previous similar message [ 4464.140749] Lustre: Skipped 1 previous similar message [ 4469.043368] Lustre: server umount lustre-MDT0001 complete [ 4480.350089] Lustre: server umount lustre-OST0000 complete [ 4490.890458] Lustre: server umount lustre-OST0001 complete [ 4497.957513] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 4512.408454] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4552.834323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4553.043804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4553.076426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4553.171079] Lustre: lustre-MDT0000: new disk, initializing [ 4553.258893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4553.264375] Lustre: Skipped 5 previous similar messages [ 4553.278805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4558.686201] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4568.721094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.829420] Lustre: 136115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4568.897784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4568.906167] Lustre: Skipped 1 previous similar message [ 4568.979376] Lustre: lustre-MDT0001: new disk, initializing [ 4569.129234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4569.143964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4573.867306] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4584.311801] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4589.874343] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4598.118940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4598.490095] Lustre: lustre-OST0000: new disk, initializing [ 4598.495585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4598.507371] Lustre: 138082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4600.456744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4600.464565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4600.607086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4605.078383] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4615.885243] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4625.864718] Lustre: lustre-MDD0000: changelog on [ 4628.994981] Lustre: lustre-MDD0001: changelog on [ 6587.041638] Lustre: lustre-MDD0001: changelog off [ 6589.422449] Lustre: 140907:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6590.448459] Lustre: lustre-MDD0000: changelog off [ 6597.604364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6597.608272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6597.612270] LustreError: Skipped 1 previous similar message [ 6597.614345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6597.614353] Lustre: Skipped 2 previous similar messages [ 6597.654199] Lustre: Skipped 1 previous similar message [ 6601.712751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6601.718824] Lustre: Skipped 2 previous similar messages [ 6603.379940] Lustre: server umount lustre-OST0000 complete [ 6606.820763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.826036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6606.836545] Lustre: Skipped 1 previous similar message [ 6606.851627] Lustre: Skipped 1 previous similar message [ 6611.939082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.946753] Lustre: Skipped 1 previous similar message [ 6621.152145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6621.419677] Lustre: server umount lustre-MDT0000 complete [ 6622.181479] LustreError: 136125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6622.192261] LustreError: 136125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6625.130800] LustreError: 136106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982597 with bad export cookie 10010375900261822693 [ 6625.134284] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6625.150884] LustreError: 136106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6625.514715] Lustre: server umount lustre-MDT0001 complete [ 6642.813388] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6646.383463] Key type lgssc unregistered [ 6646.676389] LNet: 147251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6646.686284] LNetError: 147251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6646.711247] LNet: Removed LNI 192.168.202.136@tcp [ 6647.614157] Key type .llcrypt unregistered [ 6647.616242] Key type ._llcrypt unregistered [ 6667.969319] Key type ._llcrypt registered [ 6667.971615] Key type .llcrypt registered [ 6668.081466] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 6682.894696] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6683.803441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6683.923932] alg: No test for adler32 (adler32-zlib) [ 6684.900589] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 6685.120873] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6686.793796] Key type lgssc registered [ 6688.092570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6720.746102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6720.776091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6721.999449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6722.040405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6722.128638] Lustre: lustre-MDT0000: new disk, initializing [ 6722.247220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6722.264532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6726.979814] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6736.788584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.906918] Lustre: 150454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6736.975436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6736.983185] Lustre: Skipped 1 previous similar message [ 6737.083679] Lustre: lustre-MDT0001: new disk, initializing [ 6737.215323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6737.262102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6737.269756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6742.002487] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6752.069492] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6758.681284] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6766.944910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6767.195791] Lustre: lustre-OST0000: new disk, initializing [ 6767.201364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6767.210964] Lustre: 152420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6767.334150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.467320] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6774.331466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6774.339956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6774.411084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6782.068745] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.712965] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_module obdecho/obdecho [ 6790.819865] LustreError: 153423:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6790.826903] LustreError: 153423:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6792.663593] LustreError: 153519:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6809.057816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6809.073863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.098534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6810.594431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6810.597209] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.187394] Lustre: server umount lustre-MDT0000 complete [ 6817.646796] LustreError: 150445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982790 with bad export cookie 8005434546700279502 [ 6817.652475] LustreError: 150445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6817.653185] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6817.946649] Lustre: server umount lustre-MDT0001 complete [ 6831.792808] Lustre: server umount lustre-OST0000 complete [ 6854.102203] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 6866.145095] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6907.014861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.437926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6907.461064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6907.523782] Lustre: lustre-MDT0000: new disk, initializing [ 6907.635162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6907.675473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6912.215868] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6925.382697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6925.522607] Lustre: 157236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6925.589831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6925.593626] Lustre: Skipped 1 previous similar message [ 6925.684879] Lustre: lustre-MDT0001: new disk, initializing [ 6925.829749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6925.865855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6925.887854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6930.490405] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6940.484155] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6947.337673] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6954.811040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6955.066763] Lustre: lustre-OST0000: new disk, initializing [ 6955.070909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6955.076906] Lustre: 159203:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6955.158827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.808136] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6963.789673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6963.804741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6963.841467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6971.288268] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6977.212090] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6977.553574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6983.040934] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6983.336361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6989.794276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6989.807965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6989.813337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6989.818053] Lustre: Skipped 2 previous similar messages [ 6989.834134] Lustre: Skipped 3 previous similar messages [ 6991.749625] Lustre: server umount lustre-OST0000 complete [ 6997.473683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6997.488854] LustreError: Skipped 1 previous similar message [ 6997.492839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6997.511057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.156149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.160794] Lustre: Skipped 2 previous similar messages [ 7010.273993] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7010.743458] Lustre: server umount lustre-MDT0000 complete [ 7014.369926] LustreError: 158459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7014.389165] LustreError: 158459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7015.095293] LustreError: 157228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982987 with bad export cookie 8005434546700280461 [ 7015.112952] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7015.929801] Lustre: server umount lustre-MDT0001 complete [ 7020.489376] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:09:50 (1786982990) [ 7022.468468] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7024.699903] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:09:55 (1786982995) [ 7032.623663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7033.159735] LustreError: 161624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7033.222522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.241492] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7038.441994] LustreError: 161625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.567989] LustreError: 161624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7047.035734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.299241] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7057.862971] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7063.269200] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7069.420607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7069.822503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.825411] Lustre: Skipped 1 previous similar message [ 7075.953371] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7082.687035] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7088.615342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7088.623550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7088.632146] Lustre: Skipped 1 previous similar message [ 7088.641148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7088.646684] Lustre: Skipped 2 previous similar messages [ 7093.994224] Lustre: server umount lustre-MDT0000 complete [ 7094.753749] LustreError: 161624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.769718] LustreError: 161624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7097.665189] LustreError: 162359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983070 with bad export cookie 8005434546700281343 [ 7097.673729] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7097.676827] LustreError: 162359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7099.877144] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7099.898737] Lustre: Skipped 2 previous similar messages [ 7102.887467] LustreError: 161619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7102.905701] LustreError: 161619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7104.488628] Lustre: server umount lustre-MDT0001 complete [ 7111.634756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7127.581660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.092478] LustreError: 165463:0:(llog.c:1655:llog_backup()) MGC192.168.202.136@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7128.101673] Lustre: 165463:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.136@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7129.749571] LustreError: 165468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7129.767986] LustreError: 165468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7163.361579] LustreError: 165469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7163.382632] LustreError: 165469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 7188.971478] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7188.977409] Lustre: 165463:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.136@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7228.385023] LustreError: 165469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7228.419684] LustreError: 165469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 7229.958486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7235.059526] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7236.323418] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7247.575029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7258.603050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7268.836827] LustreError: 148673:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002b5b25a4 x1873787126682112/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7274.389804] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7274.495390] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7283.828858] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7285.506322] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7293.015102] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7295.359104] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7319.367638] Lustre: server umount lustre-MDT0000 complete [ 7320.550495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7320.554081] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.587446] Lustre: Skipped 1 previous similar message [ 7323.597648] LustreError: 161605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983296 with bad export cookie 8005434546700287804 [ 7323.607411] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7323.610770] LustreError: 161605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7324.001403] Lustre: server umount lustre-MDT0001 complete [ 7339.368955] Lustre: server umount lustre-OST0000 complete [ 7361.921692] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 7373.531596] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7410.568966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7410.982960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7411.051142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7411.203315] Lustre: lustre-MDT0000: new disk, initializing [ 7411.289612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7411.301061] Lustre: Skipped 1 previous similar message [ 7411.340392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7416.188445] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7427.879904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7428.056576] Lustre: 170574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7428.279422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7432.154772] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7441.780084] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7447.412769] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7454.282336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7454.498156] Lustre: lustre-OST0000: new disk, initializing [ 7454.500750] Lustre: Skipped 1 previous similar message [ 7454.503645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7454.510020] Lustre: Skipped 2 previous similar messages [ 7454.513960] Lustre: 172545:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7460.159636] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7460.937502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7460.947833] Lustre: Skipped 1 previous similar message [ 7460.960488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7461.048722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7469.083916] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7473.867764] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7474.048986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7478.294243] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7478.493866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7481.825162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7481.827942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7481.841486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7481.846788] Lustre: Skipped 2 previous similar messages [ 7481.862707] Lustre: Skipped 9 previous similar messages [ 7486.553956] Lustre: server umount lustre-OST0000 complete [ 7496.830801] Lustre: server umount lustre-MDT0000 complete [ 7497.189775] LustreError: 170585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.216737] LustreError: 170585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 7501.166522] LustreError: 170565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983474 with bad export cookie 8005434546700304618 [ 7501.177182] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.178680] LustreError: 170565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7501.751411] Lustre: server umount lustre-MDT0001 complete [ 7510.794148] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:18:01 (1786983481) [ 7517.485380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7522.400712] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7531.942458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.863625] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7545.012913] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7551.336154] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7558.629274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7564.948360] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7573.514686] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7586.397847] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 12:19:17 (1786983557) [ 7592.931778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.935906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7592.957072] Lustre: Skipped 2 previous similar messages [ 7592.958553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7592.969849] LustreError: Skipped 1 previous similar message [ 7592.988400] Lustre: Skipped 3 previous similar messages [ 7596.620262] Lustre: server umount lustre-OST0000 complete [ 7606.793235] Lustre: server umount lustre-MDT0000 complete [ 7610.868579] LustreError: 174937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983583 with bad export cookie 8005434546700305500 [ 7610.877544] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7610.879789] LustreError: 174937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7611.313711] Lustre: server umount lustre-MDT0001 complete [ 7621.693543] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7624.455425] Key type lgssc unregistered [ 7624.730637] LNet: 178687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7624.743189] LNetError: 178687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7625.766709] LNet: Removed LNI 192.168.202.136@tcp [ 7626.547716] Key type .llcrypt unregistered [ 7626.549467] Key type ._llcrypt unregistered [ 7645.232749] Key type ._llcrypt registered [ 7645.235181] Key type .llcrypt registered [ 7645.324446] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7646.309556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7646.502141] alg: No test for adler32 (adler32-zlib) [ 7647.570790] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 7647.862619] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7649.592323] Key type lgssc registered [ 7650.935597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7659.507568] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7659.519881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7675.108706] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7680.225369] LustreError: 179932:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 7685.344769] LustreError: 179932:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7685.363662] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7685.374654] LustreError: MGC192.168.202.136@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7685.391237] LustreError: 179932:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7685.417489] LustreError: 179932:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7685.423885] LustreError: 179932:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7685.546118] Lustre: server umount lustre-OST0000 complete [ 7685.552169] LustreError: 179932:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7695.054698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7696.567619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.441262] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7707.116294] LustreError: 180432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7707.139991] LustreError: 180432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7711.499771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7712.118694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7717.121782] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7725.093335] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7732.163831] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7738.953754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7739.116463] LustreError: 182249:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7739.126386] Lustre: 182249:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.136@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7739.322482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7745.620351] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7755.097963] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7758.308622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7758.312657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.330784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.899129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.905220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.907137] Lustre: Skipped 1 previous similar message [ 7761.933566] Lustre: Skipped 1 previous similar message [ 7763.593532] Lustre: server umount lustre-MDT0000 complete [ 7767.016121] LustreError: 180431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7767.046510] LustreError: 180431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7767.125749] LustreError: 180413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983739 with bad export cookie 1717795913932911294 [ 7767.140074] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7767.515861] Lustre: server umount lustre-MDT0001 complete [ 7782.485365] Lustre: server umount lustre-OST0000 complete [ 7796.262785] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 7808.904494] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7850.955486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7851.236882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7851.258950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7851.383353] Lustre: lustre-MDT0000: new disk, initializing [ 7851.534583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7851.555801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7857.458043] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7867.770790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7867.882420] Lustre: 185774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7867.930693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7867.932970] Lustre: Skipped 1 previous similar message [ 7867.988493] Lustre: lustre-MDT0001: new disk, initializing [ 7868.058486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7868.082214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7868.095128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7871.910890] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7881.616623] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7889.050230] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7898.091184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7898.403801] Lustre: lustre-OST0000: new disk, initializing [ 7898.407345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7898.420068] Lustre: 187743:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7898.532642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7904.362513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7904.374653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7904.546928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7907.328342] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7919.355155] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7925.702082] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7925.931377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7930.637837] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7930.835765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7935.457348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7935.463857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.465569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7935.478641] Lustre: Skipped 1 previous similar message [ 7935.491250] Lustre: Skipped 2 previous similar messages [ 7938.827887] Lustre: server umount lustre-OST0000 complete [ 7945.185086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7945.195491] LustreError: Skipped 1 previous similar message [ 7945.203475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7945.225576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7948.941477] Lustre: server umount lustre-MDT0000 complete [ 7950.817284] LustreError: 185787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7950.833900] LustreError: 185787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7952.485285] LustreError: 186570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983925 with bad export cookie 1717795913932912015 [ 7952.490765] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7952.493963] LustreError: 186570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7952.824456] Lustre: server umount lustre-MDT0001 complete [ 7962.070205] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:25:32 (1786983932) [ 7980.186193] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7982.976762] Key type lgssc unregistered [ 7983.201605] LNet: 190694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7983.207151] LNetError: 190694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7983.217842] LNet: Removed LNI 192.168.202.136@tcp [ 7983.895223] Key type .llcrypt unregistered [ 7983.897122] Key type ._llcrypt unregistered [ 8006.482397] Key type ._llcrypt registered [ 8006.484943] Key type .llcrypt registered [ 8006.578882] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8007.883490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8007.936291] alg: No test for adler32 (adler32-zlib) [ 8008.980749] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 8009.212948] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8010.912168] Key type lgssc registered [ 8012.039814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8019.960259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8019.974863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.402830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8026.150604] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8031.714928] LustreError: 191915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8031.734715] LustreError: 191915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8034.522962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.799944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8039.436368] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8048.591969] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8054.038695] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8062.406653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8062.644133] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8062.648452] LustreError: 193777:0:(llog.c:1696:llog_backup()) MGC192.168.202.136@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8062.660317] Lustre: 193777:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.136@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8062.837322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8069.419963] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8077.605379] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8080.865776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8080.876721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.900525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8084.449992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8084.475150] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.502159] Lustre: Skipped 1 previous similar message [ 8086.801417] Lustre: server umount lustre-MDT0000 complete [ 8089.571994] LustreError: 191910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8089.593056] LustreError: 191910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8091.114622] LustreError: 191896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984063 with bad export cookie 14644778529843007667 [ 8091.137644] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8091.141860] LustreError: 191896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8091.655743] Lustre: server umount lustre-MDT0001 complete [ 8105.503727] Lustre: server umount lustre-OST0000 complete [ 8117.691600] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 8136.602651] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8182.944931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8183.213547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8183.272957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8183.396669] Lustre: lustre-MDT0000: new disk, initializing [ 8183.503167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8183.520422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8188.599558] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8199.278348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.392280] Lustre: 197328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8199.475806] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8199.477665] Lustre: Skipped 1 previous similar message [ 8199.487963] LustreError: 198110:0:(llog.c:1696:llog_backup()) MGC192.168.202.136@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8199.495380] LustreError: 198110:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8199.503088] LustreError: 197338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8199.504522] Lustre: 198110:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.136@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8199.545572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8199.553326] Lustre: Skipped 1 previous similar message [ 8199.636736] Lustre: lustre-MDT0001: new disk, initializing [ 8199.738295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8199.778641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8199.791827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8205.389978] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8217.049822] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8225.860563] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8233.427346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8233.712166] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8233.718810] Lustre: Skipped 1 previous similar message [ 8233.725215] LustreError: 199272:0:(llog.c:1696:llog_backup()) MGC192.168.202.136@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8233.737987] LustreError: 199272:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8233.745445] Lustre: 199272:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.136@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8233.759659] Lustre: 199272:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8233.790695] Lustre: lustre-OST0000: new disk, initializing [ 8233.795487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8233.813497] Lustre: 199295:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8233.972991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8241.333768] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8242.320119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8242.338433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8242.469585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8252.018337] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8257.968682] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8258.265639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8263.949652] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8264.158339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8268.261614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8268.261970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.284913] LustreError: Skipped 1 previous similar message [ 8268.288702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8268.307176] Lustre: Skipped 1 previous similar message [ 8268.337563] Lustre: Skipped 3 previous similar messages [ 8272.386681] Lustre: server umount lustre-OST0000 complete [ 8278.501064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.511329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8278.537732] Lustre: Skipped 1 previous similar message [ 8278.566929] Lustre: Skipped 1 previous similar message [ 8283.300184] Lustre: server umount lustre-MDT0000 complete [ 8283.627690] LustreError: 200635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8283.670876] LustreError: 200635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8288.324779] LustreError: 197319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984261 with bad export cookie 14644778529843008388 [ 8288.327068] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8288.335048] LustreError: 197319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8288.980984] Lustre: server umount lustre-MDT0001 complete [ 8299.647912] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:31:10 (1786984270) [ 8317.905946] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8329.980287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8330.354185] LustreError: 202426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.396969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8335.089951] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8340.962659] LustreError: 202425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8340.996483] LustreError: 202425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8345.598825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8351.870924] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8355.003568] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8362.787509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8363.022957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8363.034864] Lustre: Skipped 1 previous similar message [ 8369.713301] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8380.073530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8380.356534] Lustre: lustre-OST0001: new disk, initializing [ 8380.362197] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8380.380935] Lustre: 204732:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8388.241110] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8390.277550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8390.297768] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8390.485775] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8401.543930] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8409.441447] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8412.268579] Lustre: DEBUG MARKER: ADD OST3 [ 8418.336216] loop: module loaded [ 8418.472021] loop0: detected capacity change from 0 to 204800000 [ 8418.745553] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8418.757386] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8418.772307] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8427.025776] Lustre: DEBUG MARKER: STOP OST3 [ 8427.055599] loop0: detected capacity change from 0 to 204800000 [ 8436.475466] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8436.818600] Lustre: lustre-OST0003: new disk, initializing [ 8436.831518] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8436.846454] Lustre: 206806:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8437.102418] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8437.118330] Lustre: Skipped 1 previous similar message [ 8438.314917] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8438.580167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8438.603786] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8438.699147] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8440.353915] LustreError: 206872:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8441.398830] Lustre: Failing over lustre-OST0003 [ 8441.702933] Lustre: server umount lustre-OST0003 complete [ 8443.888411] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8443.926197] LustreError: 203911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.931364] Lustre: Skipped 1 previous similar message [ 8449.699164] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8462.111623] Lustre: DEBUG MARKER: START OST3 again [ 8470.361438] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8470.660923] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8472.359890] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8472.530672] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8472.534127] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8477.983196] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8499.883200] Lustre: Failing over lustre-OST0003 [ 8500.249506] Lustre: server umount lustre-OST0003 complete [ 8501.243521] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8501.264534] LustreError: 203911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8501.284914] LustreError: 203911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 8521.706290] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8521.716058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8521.725480] Lustre: Skipped 4 previous similar messages [ 8521.725862] Lustre: Skipped 1 previous similar message [ 8527.420672] Lustre: server umount lustre-MDT0000 complete [ 8532.063349] LustreError: 203160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984504 with bad export cookie 14644778529843009270 [ 8532.075213] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8532.077363] LustreError: 203160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8537.068845] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8537.079881] Lustre: Skipped 5 previous similar messages [ 8538.696145] Lustre: server umount lustre-MDT0001 complete [ 8542.955669] Lustre: server umount lustre-OST0000 complete [ 8547.365724] Lustre: server umount lustre-OST0001 complete [ 8555.710315] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 8568.849844] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8616.746618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.324750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8617.435051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8617.617952] Lustre: lustre-MDT0000: new disk, initializing [ 8617.821993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8617.830608] Lustre: Skipped 1 previous similar message [ 8617.883711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8621.571448] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8634.752954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.085915] Lustre: 212352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8635.168973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8635.172497] Lustre: Skipped 1 previous similar message [ 8635.285151] Lustre: lustre-MDT0001: new disk, initializing [ 8635.458168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8635.466288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8640.109521] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8650.195145] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8656.764625] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8664.173482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8664.433440] Lustre: 214321:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8666.313265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8666.465496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8671.776972] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8682.055646] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8687.297454] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8687.574058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8693.327410] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8693.550802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8697.317661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8697.318318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8697.332655] Lustre: Skipped 2 previous similar messages [ 8697.342390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8697.368293] Lustre: Skipped 1 previous similar message [ 8701.809148] Lustre: server umount lustre-OST0000 complete [ 8707.042221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8720.352131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8720.490795] Lustre: server umount lustre-MDT0000 complete [ 8722.914967] LustreError: 213245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8722.941652] LustreError: 213245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 8725.924918] LustreError: 212346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984698 with bad export cookie 14644778529843011566 [ 8725.934603] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8725.946210] LustreError: 212346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8726.684412] Lustre: server umount lustre-MDT0001 complete [ 8731.517331] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:38:21 (1786984701) [ 8750.827409] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 8761.512184] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8802.593662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8802.973194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8803.067929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8803.074856] Lustre: Skipped 1 previous similar message [ 8803.221933] Lustre: lustre-MDT0000: new disk, initializing [ 8803.227804] Lustre: Skipped 1 previous similar message [ 8803.359051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8803.373166] Lustre: Skipped 1 previous similar message [ 8809.424916] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8820.514308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.710960] Lustre: 218941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8821.069379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8825.327407] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8835.429218] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8841.498431] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8849.719801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8850.105516] Lustre: 220906:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8852.249332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8858.446701] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8867.880630] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8872.386992] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8872.598056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8877.688634] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8877.914347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8883.171237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8883.176373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8883.183836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8883.189735] Lustre: Skipped 4 previous similar messages [ 8883.217279] Lustre: Skipped 7 previous similar messages [ 8886.001803] Lustre: server umount lustre-OST0000 complete [ 8892.900543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8892.905588] LustreError: Skipped 1 previous similar message [ 8904.160191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8904.323519] Lustre: server umount lustre-MDT0000 complete [ 8907.849942] LustreError: 218932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984880 with bad export cookie 14644778529843012448 [ 8907.855133] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8907.860747] LustreError: 218932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8908.207706] Lustre: server umount lustre-MDT0001 complete [ 8916.640990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8916.992699] LustreError: 223125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8917.062232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8917.066751] Lustre: Skipped 5 previous similar messages [ 8921.839842] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8931.631988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.891724] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8944.442902] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8951.574467] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8958.015112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8964.592397] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8971.850307] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8986.085739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8986.102744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8986.113527] Lustre: Skipped 7 previous similar messages [ 8997.345162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8997.503460] Lustre: server umount lustre-OST0000 complete [ 9008.733151] Lustre: server umount lustre-MDT0000 complete [ 9013.270966] LustreError: 223105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984986 with bad export cookie 14644778529843013330 [ 9013.308336] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9013.878645] Lustre: server umount lustre-MDT0001 complete [ 9111.275724] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9114.194956] Key type lgssc unregistered [ 9114.621323] LNet: 226788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9114.627333] LNetError: 226788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9114.637077] LNet: Removed LNI 192.168.202.136@tcp [ 9115.581075] Key type .llcrypt unregistered [ 9115.582532] Key type ._llcrypt unregistered [ 9148.550036] Key type ._llcrypt registered [ 9148.551546] Key type .llcrypt registered [ 9148.690845] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 9167.729921] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9169.348169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9169.446541] alg: No test for adler32 (adler32-zlib) [ 9170.544761] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [ 9170.811707] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9172.512177] Key type lgssc registered [ 9173.783205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9213.744903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9213.791475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9215.125270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9215.174182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9215.282549] Lustre: lustre-MDT0000: new disk, initializing [ 9215.399855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9215.437606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9220.555174] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9232.250282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9232.382209] Lustre: 230144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9232.431472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9232.434277] Lustre: Skipped 1 previous similar message [ 9232.502440] Lustre: lustre-MDT0001: new disk, initializing [ 9232.621123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9232.665044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9232.689505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9237.395746] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9248.252551] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9255.694086] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9262.201776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9262.613850] Lustre: lustre-OST0000: new disk, initializing [ 9262.621136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9262.631676] Lustre: 232111:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9262.733457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9269.236862] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9270.886456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9270.900778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9270.991568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9278.737758] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9283.751156] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9284.011369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9289.440272] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9289.636472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9291.748294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9291.758599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9291.764961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9291.765433] Lustre: Skipped 1 previous similar message [ 9291.779673] Lustre: Skipped 1 previous similar message [ 9296.888361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9296.898620] Lustre: Skipped 1 previous similar message [ 9297.904255] Lustre: server umount lustre-OST0000 complete [ 9301.995273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9302.010633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9305.057918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9305.069340] Lustre: Skipped 2 previous similar messages [ 9311.201410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9311.209252] Lustre: Skipped 3 previous similar messages [ 9316.321221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9316.464509] Lustre: server umount lustre-MDT0000 complete [ 9319.990072] LustreError: 232110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985292 with bad export cookie 3949132092260572484 [ 9319.994679] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9319.999549] LustreError: 232110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9320.415519] Lustre: server umount lustre-MDT0001 complete [ 9341.212057] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9351.546278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9352.186091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9357.280990] LustreError: 235045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9357.296587] LustreError: 235045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9357.474233] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9361.384135] LustreError: 235046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9366.505431] LustreError: 235045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9366.650089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9366.978773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9372.029204] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9375.680328] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9382.837358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9383.295290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9390.466495] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9399.937654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9400.103698] Lustre: lustre-OST0001: new disk, initializing [ 9400.107080] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9400.112075] Lustre: 237352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9400.210862] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9406.946035] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9407.599424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9407.611601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9407.696621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9419.293650] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9427.632723] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9444.113810] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:50:14 (1786985414) [ 9448.941255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9448.951687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9448.953010] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9448.953017] LustreError: Skipped 1 previous similar message [ 9448.965667] Lustre: Skipped 4 previous similar messages [ 9448.974706] Lustre: Skipped 4 previous similar messages [ 9452.600905] Lustre: server umount lustre-MDT0000 complete [ 9454.051136] LustreError: 235045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9454.085676] LustreError: 235045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9456.486717] LustreError: 238122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985429 with bad export cookie 3949132092260573359 [ 9456.502150] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9456.506507] LustreError: 238122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9456.869453] Lustre: server umount lustre-MDT0001 complete [ 9471.688314] Lustre: server umount lustre-OST0000 complete [ 9476.069823] Lustre: 228369:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786985432/real 1786985432] req@ffff94f54c181180 x1873789733420288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786985448 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9476.111915] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9476.397979] Lustre: server umount lustre-OST0001 complete [ 9484.227866] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 9495.220864] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9531.790455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9532.258838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9532.326252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9532.458697] Lustre: lustre-MDT0000: new disk, initializing [ 9532.531030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.571978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9538.150658] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9551.286221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.481764] Lustre: 241300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9551.558387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9551.563258] Lustre: Skipped 1 previous similar message [ 9551.636301] Lustre: lustre-MDT0001: new disk, initializing [ 9551.790152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9551.796479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9557.273983] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9567.751636] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9574.741509] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9582.802597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9583.195874] Lustre: lustre-OST0000: new disk, initializing [ 9583.199803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9583.206964] Lustre: 243269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9585.134622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9585.150146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9585.353029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9590.398374] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9600.516363] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9606.905277] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9607.227157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9613.394405] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9613.769883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9615.846030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9615.849467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9615.872095] Lustre: Skipped 1 previous similar message [ 9615.881988] Lustre: Skipped 2 previous similar messages [ 9621.948837] Lustre: server umount lustre-OST0000 complete [ 9626.085181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9632.354196] Lustre: server umount lustre-MDT0000 complete [ 9636.324219] LustreError: 241311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9636.354479] LustreError: 241311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9636.408333] LustreError: 241291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985609 with bad export cookie 3949132092260575501 [ 9636.415827] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9636.429536] LustreError: 241291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9637.010198] Lustre: server umount lustre-MDT0001 complete [ 9647.065866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9647.787484] LustreError: 245488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9647.924649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9647.939824] Lustre: Skipped 2 previous similar messages [ 9654.900586] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9665.479136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9670.309475] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9678.192715] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9684.045967] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9691.452777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.728517] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9706.287338] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9851.874470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9851.878778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9851.886714] Lustre: Skipped 1 previous similar message [ 9851.894358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9851.899245] Lustre: Skipped 6 previous similar messages [ 9853.929858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9865.184242] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9865.508345] Lustre: server umount lustre-OST0000 complete [ 9870.817731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9875.881561] Lustre: server umount lustre-MDT0000 complete [ 9879.527604] LustreError: 245489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.546571] LustreError: 245489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9879.998526] LustreError: 245469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985852 with bad export cookie 3949132092260576383 [ 9880.017505] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9880.029563] LustreError: 245469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9880.571724] Lustre: server umount lustre-MDT0001 complete [ 9974.590632] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9977.232883] Key type lgssc unregistered [ 9977.511822] LNet: 249100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9977.526742] LNetError: 249100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9978.601125] LNet: Removed LNI 192.168.202.136@tcp [ 9979.438262] Key type .llcrypt unregistered [ 9979.441674] Key type ._llcrypt unregistered [10005.559915] Key type ._llcrypt registered [10005.561922] Key type .llcrypt registered [10005.656967] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [10021.395524] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10022.663743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10022.743046] alg: No test for adler32 (adler32-zlib) [10023.955213] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [10024.234331] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10025.904189] Key type lgssc registered [10026.987324] Lustre: Echo OBD driver; http://www.lustre.org/ [10071.822610] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10082.865864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10082.917587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.214930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10084.240375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10084.299469] Lustre: lustre-MDT0000: new disk, initializing [10084.341654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10084.356512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10089.196631] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10100.317635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10100.533691] Lustre: 253174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10100.662992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10100.672182] Lustre: Skipped 1 previous similar message [10100.804558] Lustre: lustre-MDT0001: new disk, initializing [10101.007554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10101.099950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10101.123209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10106.493727] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10111.521636] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10119.213301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10119.730615] Lustre: lustre-OST0000: new disk, initializing [10119.745636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10119.758323] Lustre: 254813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10119.958422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10127.699812] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10128.969974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10128.982368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10129.111794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10137.853393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10138.114948] Lustre: lustre-OST0001: new disk, initializing [10138.122841] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10138.136489] Lustre: 255686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10138.279441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10143.897683] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10146.959697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10146.967875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10147.085344] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10154.888544] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10160.751972] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10172.901151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10172.922768] Lustre: Skipped 1 previous similar message [10172.925238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10178.022712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10178.040585] Lustre: Skipped 6 previous similar messages [10178.294661] Lustre: server umount lustre-MDT0000 complete [10181.951034] LustreError: 253164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986154 with bad export cookie 11286724478452197674 [10181.955102] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10181.958246] LustreError: 253164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10182.392864] Lustre: server umount lustre-MDT0001 complete [10196.712994] Lustre: server umount lustre-OST0000 complete [10199.520227] Lustre: 250679:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786986156/real 1786986156] req@ffff94f54b84f800 x1873790628867328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786986172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10199.555888] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10199.591606] Lustre: Skipped 2 previous similar messages [10200.696677] Lustre: server umount lustre-OST0001 complete [10213.877916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10222.958807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10231.187705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10241.276879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10262.650832] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10273.564753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10273.586306] Lustre: lustre-MDT0000: reset Object Index mappings [10274.267296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10279.038328] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10279.403299] LustreError: 260693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10279.441299] LustreError: 260693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10284.515557] LustreError: 260694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10287.371964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10287.397064] Lustre: lustre-MDT0001: reset Object Index mappings [10287.924588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10293.256294] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10296.905246] Lustre: 261835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10304.959334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10305.299928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10312.498027] LustreError: 262189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10313.691990] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10317.819322] LustreError: 262190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10319.955724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10324.972533] LustreError: 262500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.008354] LustreError: 262500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10325.141813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10330.612036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10332.816217] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10340.271826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10344.397606] Lustre: 263707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10349.537826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10349.546459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10349.565199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10351.682931] Lustre: server umount lustre-MDT0000 complete [10355.434281] LustreError: 260674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986328 with bad export cookie 11286724478452204464 [10355.452934] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10355.456914] LustreError: 260674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10355.850934] Lustre: server umount lustre-MDT0001 complete [10369.273760] Lustre: server umount lustre-OST0000 complete [10372.576280] Lustre: 250679:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786986329/real 1786986329] req@ffff94f547c37b80 x1873790628918784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786986345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10372.596710] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10372.612353] Lustre: Skipped 3 previous similar messages [10373.138620] Lustre: server umount lustre-OST0001 complete [10393.937458] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:06:04 (1786986364) [10411.555320] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [10422.471939] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10462.958712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10463.250577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10463.289801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10463.360351] Lustre: lustre-MDT0000: new disk, initializing [10463.471057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10463.478655] Lustre: Skipped 1 previous similar message [10463.505649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10468.620661] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10482.147930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10482.258523] Lustre: 267958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10482.380781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10482.388173] Lustre: Skipped 1 previous similar message [10482.519383] Lustre: lustre-MDT0001: new disk, initializing [10482.690668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10482.714303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10487.689355] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10498.006609] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10503.852239] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10510.885866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10511.167806] Lustre: lustre-OST0000: new disk, initializing [10511.171608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10511.181333] Lustre: 269929:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10512.283992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10512.305224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10512.513420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10518.106951] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10527.382486] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10532.151517] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10532.383864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10536.976812] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10537.122509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10543.074641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10543.092957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10543.101088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10543.107573] Lustre: Skipped 1 previous similar message [10543.118196] Lustre: Skipped 5 previous similar messages [10545.455865] Lustre: server umount lustre-OST0000 complete [10553.334829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.353414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10553.353491] Lustre: Skipped 1 previous similar message [10553.366647] Lustre: Skipped 1 previous similar message [10557.338638] Lustre: server umount lustre-MDT0000 complete [10558.442348] LustreError: 267966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10558.461887] LustreError: 267966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10561.564316] LustreError: 269928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986534 with bad export cookie 11286724478452206557 [10561.576521] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10561.591675] LustreError: 269928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10562.122839] Lustre: server umount lustre-MDT0001 complete [10584.505484] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10595.469887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10595.992953] LustreError: 272864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10596.072347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10596.082097] Lustre: Skipped 2 previous similar messages [10600.517275] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10609.265992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10614.259530] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10617.126303] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10623.798634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10631.023225] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10639.398550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10639.586330] Lustre: lustre-OST0001: new disk, initializing [10639.590282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10639.595633] Lustre: 275170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10640.866911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10640.877809] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10641.062567] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10646.215816] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10656.539936] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10661.233496] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10666.598341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10666.978989] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10676.197617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10676.204381] LustreError: Skipped 1 previous similar message [10676.211179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10676.222704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10680.312056] Lustre: server umount lustre-MDT0000 complete [10681.835366] LustreError: 272860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10681.867962] LustreError: 272860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10684.153373] LustreError: 272845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986657 with bad export cookie 11286724478452207439 [10684.155130] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10684.162268] LustreError: 272845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10684.559308] Lustre: server umount lustre-MDT0001 complete [10698.829716] Lustre: server umount lustre-OST0000 complete [10702.310687] Lustre: 250681:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786986659/real 1786986659] req@ffff94f54c2b1180 x1873790629030016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786986675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10703.580459] Lustre: server umount lustre-OST0001 complete [10723.122517] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10734.151887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.792936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10734.797853] Lustre: Skipped 3 previous similar messages [10740.758527] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10749.417907] LustreError: 278228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10749.430947] LustreError: 278228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10752.375217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10758.453254] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10761.668657] Lustre: 279369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10769.698158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10776.929526] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10779.390511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10786.663679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10792.458958] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10794.315311] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10803.330034] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10807.558695] Lustre: 281244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10813.319115] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10813.345188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10824.072380] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:13:14 (1786986794) [10840.035319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10840.055218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10840.084046] Lustre: Skipped 4 previous similar messages [10840.088466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10840.099202] Lustre: Skipped 3 previous similar messages [10853.344514] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10853.821833] Lustre: server umount lustre-MDT0000 complete [10858.646433] LustreError: 278208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986831 with bad export cookie 11286724478452214215 [10858.682785] LustreError: 278208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10858.685200] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10865.455697] Lustre: server umount lustre-MDT0001 complete [10875.929496] Lustre: server umount lustre-OST0000 complete [10879.764379] Lustre: server umount lustre-OST0001 complete [10887.211486] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [10898.770400] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10949.013848] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10960.948188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.196611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10961.247819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10961.339495] Lustre: lustre-MDT0000: new disk, initializing [10961.446485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10966.504236] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10979.127520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10979.344195] Lustre: 285492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10979.764028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10985.559460] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11000.084141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11000.356544] Lustre: 287125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11000.441397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11000.444546] Lustre: Skipped 5 previous similar messages [11002.175994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11009.824396] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11022.762633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11022.970479] Lustre: 287997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11024.367124] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11024.374987] Lustre: Skipped 1 previous similar message [11024.486571] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11030.060548] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11040.466283] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11044.078364] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11044.082718] Lustre: Skipped 1 previous similar message [11061.895425] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:17:12 (1786987032) [11065.320439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11065.334499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11065.341040] Lustre: Skipped 6 previous similar messages [11065.357560] Lustre: Skipped 14 previous similar messages [11066.992529] Lustre: server umount lustre-MDT0000 complete [11070.448940] LustreError: 287589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11070.458838] LustreError: 287589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [11071.617296] LustreError: 285484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987044 with bad export cookie 11286724478452221656 [11071.629246] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11071.858920] Lustre: server umount lustre-MDT0001 complete [11086.970265] Lustre: server umount lustre-OST0000 complete [11102.061710] Lustre: server umount lustre-OST0001 complete [11111.061665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11117.197734] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11127.591129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11133.288802] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11140.842896] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11147.638678] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11156.411130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11163.525374] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11171.295157] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11191.366643] loop1: detected capacity change from 0 to 204800000 [11191.614690] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11191.634286] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11191.644051] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11197.095056] loop1: detected capacity change from 0 to 204800000 [11202.872577] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11203.010153] LustreError: 290236:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs16497: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11203.028169] LustreError: 290236:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs16497 [11203.033832] LustreError: 290236:0:(mgs_handler.c:517:mgs_target_reg()) fs16497-MDT0001: Error checking target index: rc = -13 [11203.040649] LustreError: fs16497-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11203.052620] LustreError: 293962:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11203.062766] LustreError: 293962:0:(tgt_mount.c:2129:server_put_super()) no obd fs16497-MDT0001 [11203.071794] LustreError: 293962:0:(tgt_mount.c:116:server_deregister_mount()) fs16497-MDT0001 not registered [11203.102689] Lustre: server umount fs16497-MDT0001 complete [11203.105405] LustreError: 293962:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11211.883193] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11212.033054] Lustre: Setting parameter fs16497-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs16497-MDT0001 [11212.089322] Lustre: srv-fs16497-MDT0001: No data found on store. Initialize space. [11212.094743] Lustre: Skipped 4 previous similar messages [11212.178452] Lustre: fs16497-MDT0001: new disk, initializing [11212.181186] Lustre: Skipped 3 previous similar messages [11212.331796] Lustre: srv-fs16497-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11212.342838] Lustre: 294430:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16497-MDT0001-mdtlov: can't create new llog: rc = -115 [11212.362489] LustreError: 294430:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16497-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11217.374209] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11224.655059] Lustre: server umount fs16497-MDT0001 complete [11236.682524] loop1: detected capacity change from 0 to 204800000 [11238.901513] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11242.281101] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11242.716531] Lustre: 295998:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16497-MDT0001-mdtlov: can't create new llog: rc = -115 [11242.736655] LustreError: 295998:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16497-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11247.165791] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11252.199293] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11252.244432] Lustre: server umount fs16497-MDT0001 complete [11262.130537] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [11262.434058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11262.474246] Lustre: Skipped 4 previous similar messages [11268.550193] Lustre: server umount lustre-MDT0000 complete [11278.202543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11278.379182] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11283.977452] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11284.136652] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11297.196186] loop1: detected capacity change from 0 to 204800000 [11302.852776] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11303.212802] Lustre: 298443:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16497-MDT0001-mdtlov: can't create new llog: rc = -115 [11303.226898] LustreError: 298443:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16497-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11307.093193] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11312.130781] Lustre: server umount fs16497-MDT0001 complete [11328.684888] LustreError: 292404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11328.707829] LustreError: 292404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 106 previous similar messages [11342.696575] Lustre: server umount lustre-OST0000 complete [11353.933383] Lustre: server umount lustre-MDT0000 complete [11357.833770] LustreError: 290227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987330 with bad export cookie 11286724478452224337 [11357.840161] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11357.852491] LustreError: 290227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11358.399796] Lustre: server umount lustre-MDT0001 complete [11370.870700] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [11374.008240] Key type lgssc unregistered [11374.410718] LNet: 300792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11374.419103] LNetError: 300792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11374.439781] LNet: Removed LNI 192.168.202.136@tcp [11375.463167] Key type .llcrypt unregistered [11375.465044] Key type ._llcrypt unregistered [11396.388578] Key type ._llcrypt registered [11396.391736] Key type .llcrypt registered [11396.508940] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [11398.023455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11398.158515] alg: No test for adler32 (adler32-zlib) [11399.303169] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [11399.517223] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [11401.162363] Key type lgssc registered [11402.227840] Lustre: Echo OBD driver; http://www.lustre.org/ [11409.856934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11409.871496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11411.388683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11416.491808] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11421.676347] LustreError: 302063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11421.718257] LustreError: 302063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11426.677185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11426.796639] LustreError: 302064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11427.111641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11432.328986] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11441.481625] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11449.073337] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11458.530805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11458.959223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11465.389855] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11471.281241] LustreError: 303894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.821205] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11476.476094] LustreError: 303895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.598109] LustreError: 304520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.634283] LustreError: 304520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11491.829130] LustreError: 303895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.852101] LustreError: 303895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11501.091977] loop1: detected capacity change from 0 to 204800000 [11501.356852] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11501.370976] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11501.395973] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11507.869537] loop1: detected capacity change from 0 to 204800000 [11509.434874] LustreError: 304520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11509.466715] LustreError: 304520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11515.192502] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11515.408765] Lustre: fs28842-OST0000: new disk, initializing [11515.411988] Lustre: srv-fs28842-OST0000: No data found on store. Initialize space. [11515.418503] Lustre: 305825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11515.492427] Lustre: fs28842-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11523.228454] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11531.985612] Lustre: server umount fs28842-OST0000 complete [11543.029861] LustreError: 303896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11543.070800] LustreError: 303896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11544.332418] loop1: detected capacity change from 0 to 204800000 [11549.913683] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11550.098406] Lustre: fs28842-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11553.261063] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11556.286742] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11558.383291] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11565.126262] loop2: detected capacity change from 0 to 204800000 [11565.391033] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11565.416268] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11565.428762] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11570.831748] loop2: detected capacity change from 0 to 204800000 [11576.321359] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11576.446456] LustreError: fs28842-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11576.465493] LustreError: fs28842-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11576.479175] LustreError: 308866:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11576.493699] LustreError: 308866:0:(tgt_mount.c:2129:server_put_super()) no obd fs28842-OST0001 [11576.498664] LustreError: 308866:0:(tgt_mount.c:116:server_deregister_mount()) fs28842-OST0001 not registered [11576.535348] Lustre: server umount fs28842-OST0001 complete [11576.538123] LustreError: 308866:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11586.165449] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11586.254383] Lustre: fs28842-OST0001: new disk, initializing [11586.264869] Lustre: srv-fs28842-OST0001: No data found on store. Initialize space. [11586.272954] Lustre: 309417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11586.352478] Lustre: fs28842-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11592.162156] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11600.023518] Lustre: server umount fs28842-OST0000 complete [11609.195551] Lustre: server umount fs28842-OST0001 complete [11609.570977] LustreError: 303895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11609.597348] LustreError: 303895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11621.478462] loop1: detected capacity change from 0 to 204800000 [11626.668326] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11626.900097] Lustre: fs28842-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11635.845641] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11646.011671] loop2: detected capacity change from 0 to 204800000 [11652.329792] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11660.444179] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11672.251779] Lustre: server umount fs28842-OST0000 complete [11681.140764] Lustre: server umount fs28842-OST0001 complete [11696.612037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11696.634067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11696.653456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11696.659638] Lustre: Skipped 1 previous similar message [11700.506764] Lustre: server umount lustre-OST0000 complete [11706.856314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11706.884942] Lustre: Skipped 1 previous similar message [11706.894622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11706.914146] Lustre: Skipped 1 previous similar message [11712.328801] Lustre: server umount lustre-MDT0000 complete [11717.476226] LustreError: 302044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987690 with bad export cookie 6374850716742768137 [11717.477582] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11717.491438] LustreError: 302044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11718.159526] Lustre: server umount lustre-MDT0001 complete [11731.861211] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [11735.163443] Key type lgssc unregistered [11735.544148] LNet: 315346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11735.551586] LNetError: 315346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11735.570188] LNet: Removed LNI 192.168.202.136@tcp [11736.437179] Key type .llcrypt unregistered [11736.439717] Key type ._llcrypt unregistered [11769.088308] Key type ._llcrypt registered [11769.090900] Key type .llcrypt registered [11769.193197] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [11783.722732] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [11784.857719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11785.044581] alg: No test for adler32 (adler32-zlib) [11786.092667] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [11786.257330] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [11787.880184] Key type lgssc registered [11789.014196] Lustre: Echo OBD driver; http://www.lustre.org/ [11830.644960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11830.746032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11832.218128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11832.274369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11832.374764] Lustre: lustre-MDT0000: new disk, initializing [11832.500670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11832.517499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11839.050645] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11854.370650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11854.472952] Lustre: 319207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11854.504451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11854.507877] Lustre: Skipped 1 previous similar message [11854.622691] Lustre: lustre-MDT0001: new disk, initializing [11854.690216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11854.721976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11854.731100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11859.232601] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11868.755942] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11874.290124] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11880.873441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11881.093977] Lustre: lustre-OST0000: new disk, initializing [11881.097516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11881.102732] Lustre: 321177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11881.196241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11887.720501] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11888.179619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11888.184889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11888.275505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11899.515564] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11901.709198] LustreError: 317378:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0001_UUID failed: -22 [11901.788620] Lustre: Mounted lustre-client - version 2.17.56_50_g3eb67fe [11902.870514] Lustre: Unmounted lustre-client [11913.699916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11913.720825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11913.739883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11914.209995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11914.222459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11914.255985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11917.282297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11917.308804] Lustre: Skipped 1 previous similar message [11918.785629] Lustre: server umount lustre-OST0000 complete [11926.500743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11926.526778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11926.552492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11932.649447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11932.656050] Lustre: Skipped 3 previous similar messages [11937.248163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11937.496729] Lustre: server umount lustre-MDT0000 complete [11941.554204] LustreError: 320003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987914 with bad export cookie 178592278148545077 [11941.562331] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11941.566069] LustreError: 320003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11942.100355] Lustre: server umount lustre-MDT0001 complete [11954.717699] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [11955.296153] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp [11958.298244] Key type lgssc unregistered [11958.730824] LNet: 323341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11958.737888] LNetError: 323341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12031.472440] LNet: Removed LNI 192.168.202.136@tcp [12032.904660] Key type .llcrypt unregistered [12032.909136] Key type ._llcrypt unregistered [12052.510885] Key type ._llcrypt registered [12052.512887] Key type .llcrypt registered [12052.630694] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [12071.461430] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12073.320762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12073.460440] alg: No test for adler32 (adler32-zlib) [12074.740042] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [12075.171762] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12077.027437] Key type lgssc registered [12078.808636] Lustre: Echo OBD driver; http://www.lustre.org/ [12118.109575] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:34:48 (1786988088) [12128.444474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12128.522345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12129.829640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12129.982886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12130.093701] Lustre: lustre-MDT0000: new disk, initializing [12130.212718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12130.228242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12138.424067] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12151.443533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12151.535533] Lustre: 326563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12151.586024] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12151.588817] Lustre: Skipped 1 previous similar message [12151.655726] Lustre: lustre-MDT0001: new disk, initializing [12151.774968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12151.841269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12151.856508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12158.334257] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12169.071726] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12175.750617] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12183.905474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12184.341363] Lustre: lustre-OST0000: new disk, initializing [12184.344972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12184.350727] Lustre: 328534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12184.471930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12190.761345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12190.774364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12190.861918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12191.405672] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12200.831584] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12221.412958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12221.421430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12221.435343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12221.921436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12221.928543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12221.938520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12225.482819] Lustre: server umount lustre-OST0000 complete [12232.169209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12232.183512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12232.188935] Lustre: Skipped 1 previous similar message [12232.205118] Lustre: Skipped 1 previous similar message [12235.532713] Lustre: server umount lustre-MDT0000 complete [12240.697203] LustreError: 327370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988213 with bad export cookie 16263875419992780703 [12240.704465] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12240.712169] LustreError: 327370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12241.550261] Lustre: server umount lustre-MDT0001 complete [12252.485663] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12255.649809] Key type lgssc unregistered [12255.919830] LNet: 330446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12255.928395] LNetError: 330446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12257.003289] LNet: Removed LNI 192.168.202.136@tcp [12258.282504] Key type .llcrypt unregistered [12258.290126] Key type ._llcrypt unregistered [12281.514549] Key type ._llcrypt registered [12281.518868] Key type .llcrypt registered [12281.634312] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12282.920810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12282.949583] alg: No test for adler32 (adler32-zlib) [12284.073572] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [12284.284589] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12285.904821] Key type lgssc registered [12287.003475] Lustre: Echo OBD driver; http://www.lustre.org/ [12295.590530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12295.612423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12297.029583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12301.361247] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12306.403665] LustreError: 331719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12306.419039] LustreError: 331719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12310.988080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12311.300811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12316.057616] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12323.179458] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12330.377118] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12338.476742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12338.967374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12346.289651] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12354.853459] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12367.925581] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12370.254388] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12377.061865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12377.071692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12377.083804] Lustre: Skipped 1 previous similar message [12377.109616] Lustre: Skipped 2 previous similar messages [12378.288338] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [12382.183493] LustreError: 331720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12382.211892] LustreError: 331720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12382.273274] Lustre: server umount lustre-MDT0000 complete [12383.417618] LustreError: 333460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12387.301992] LustreError: 333960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12387.322683] LustreError: 333960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12387.975609] LustreError: 334640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988360 with bad export cookie 14631016785584377812 [12387.991416] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12387.999245] LustreError: 334640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12388.541855] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [12392.418334] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12392.432256] Lustre: Skipped 1 previous similar message [12392.436752] LustreError: 331714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12392.440147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12392.460980] LustreError: 331714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12394.792617] Lustre: server umount lustre-MDT0001 complete [12399.430759] Lustre: server umount lustre-OST0000 complete [12409.006216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12414.352889] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12417.687730] Lustre: 335904:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.136@tcp [12420.003821] Lustre: server umount MGS complete [12429.078153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12429.494304] LustreError: 336477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12429.502948] LustreError: 336477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12429.587060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12434.291209] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12443.093890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12443.592537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12449.088950] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12457.142537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12457.347832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12464.424347] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12473.219783] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12475.924453] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12484.848895] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12487.146471] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12494.305416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12494.310828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12494.323452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12494.326061] Lustre: Skipped 1 previous similar message [12495.329391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12500.450626] Lustre: server umount lustre-OST0000 complete [12505.058274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12505.063246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12505.079606] Lustre: Skipped 1 previous similar message [12505.085251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12505.091782] Lustre: Skipped 2 previous similar messages [12518.368173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12518.841934] Lustre: server umount lustre-MDT0000 complete [12520.931268] LustreError: 336477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12520.950955] LustreError: 336477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12522.712916] LustreError: 336456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988495 with bad export cookie 14631016785584378652 [12522.722694] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12522.730510] LustreError: 336456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12523.085494] Lustre: server umount lustre-MDT0001 complete [12536.444108] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12540.021464] Key type lgssc unregistered [12540.361786] LNet: 339920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12540.367381] LNetError: 339920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12540.380524] LNet: Removed LNI 192.168.202.136@tcp [12541.520222] Key type .llcrypt unregistered [12541.522661] Key type ._llcrypt unregistered [12566.389056] Key type ._llcrypt registered [12566.393120] Key type .llcrypt registered [12566.497486] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [12584.689816] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12585.696871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12585.733129] alg: No test for adler32 (adler32-zlib) [12587.075636] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [12587.358616] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12589.104681] Key type lgssc registered [12590.425517] Lustre: Echo OBD driver; http://www.lustre.org/ [12634.704431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12634.721511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12636.038293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12636.074106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12636.191130] Lustre: lustre-MDT0000: new disk, initializing [12636.387924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12636.424531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12641.065833] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12654.302482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12654.393876] Lustre: 343084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12654.449819] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12654.458955] Lustre: Skipped 1 previous similar message [12654.523474] Lustre: lustre-MDT0001: new disk, initializing [12654.619564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12654.674645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12654.684567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12659.506494] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12669.328515] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12674.701579] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12680.776341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12681.060655] Lustre: lustre-OST0000: new disk, initializing [12681.064068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12681.067968] Lustre: 345052:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12681.158604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12687.589356] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12688.482692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12688.494948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12688.670844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12697.525416] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12702.234969] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12702.511932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12707.729523] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12707.946560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12714.469664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12714.474331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12714.483120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12714.488331] Lustre: Skipped 1 previous similar message [12719.606895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12719.610331] Lustre: Skipped 1 previous similar message [12724.192260] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12724.503845] Lustre: server umount lustre-OST0000 complete [12729.841110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12729.871233] Lustre: Skipped 1 previous similar message [12729.891379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12729.908724] Lustre: Skipped 1 previous similar message [12734.961035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12734.970748] Lustre: Skipped 1 previous similar message [12735.155617] Lustre: server umount lustre-MDT0000 complete [12739.045091] LustreError: 343075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988711 with bad export cookie 473204770015474636 [12739.053274] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12739.057439] LustreError: 343075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12739.427275] Lustre: server umount lustre-MDT0001 complete [12750.115340] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:45:20 (1786988720) [12768.406839] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [12778.777758] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12815.073670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12815.360436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12815.390573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12815.531830] Lustre: lustre-MDT0000: new disk, initializing [12815.641303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12815.658615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12820.535429] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12830.364883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12830.626917] Lustre: 349814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12830.692594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12830.696301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12830.703500] Lustre: Skipped 1 previous similar message [12830.828484] Lustre: lustre-MDT0001: new disk, initializing [12830.902354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12830.933272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12830.946505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12836.775727] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12846.606790] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12852.921482] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12860.230052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12860.594435] Lustre: lustre-OST0000: new disk, initializing [12860.598369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12860.606630] Lustre: 351786:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12860.757943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12866.613628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12866.623651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12866.716119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12867.231088] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12877.016421] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12913.130574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12913.141238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12913.151763] LustreError: Skipped 1 previous similar message [12913.168785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12913.179549] Lustre: Skipped 2 previous similar messages [12913.214021] Lustre: Skipped 1 previous similar message [12914.593246] Lustre: server umount lustre-OST0000 complete [12923.362389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12923.367861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12923.398796] Lustre: Skipped 1 previous similar message [12925.600684] Lustre: server umount lustre-MDT0000 complete [12930.098261] LustreError: 349805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988902 with bad export cookie 473204770015475518 [12930.104415] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12930.108087] LustreError: 349805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12930.607502] Lustre: server umount lustre-MDT0001 complete [12940.828866] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12944.008592] Key type lgssc unregistered [12944.277895] LNet: 353941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12944.310276] LNetError: 353941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12944.338832] LNet: Removed LNI 192.168.202.136@tcp [12945.296452] Key type .llcrypt unregistered [12945.299160] Key type ._llcrypt unregistered [13032.411782] Key type ._llcrypt registered [13032.415342] Key type .llcrypt registered [13032.498708] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13032.797386] Key type .llcrypt unregistered [13032.798666] Key type ._llcrypt unregistered [13118.263392] Key type ._llcrypt registered [13118.265390] Key type .llcrypt registered [13118.350435] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13118.690319] Key type .llcrypt unregistered [13118.693936] Key type ._llcrypt unregistered [13154.067581] Key type ._llcrypt registered [13154.072968] Key type .llcrypt registered [13154.236928] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13154.618372] Key type .llcrypt unregistered [13154.619636] Key type ._llcrypt unregistered [13187.565657] Key type ._llcrypt registered [13187.568482] Key type .llcrypt registered [13187.663801] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13187.910748] Key type .llcrypt unregistered [13187.912435] Key type ._llcrypt unregistered [13270.230650] Key type ._llcrypt registered [13270.232667] Key type .llcrypt registered [13270.317404] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13270.783505] Key type .llcrypt unregistered [13270.785519] Key type ._llcrypt unregistered [13307.546563] Key type ._llcrypt registered [13307.551409] Key type .llcrypt registered [13307.740741] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13308.087248] Key type .llcrypt unregistered [13308.088847] Key type ._llcrypt unregistered [13340.712430] Key type ._llcrypt registered [13340.716854] Key type .llcrypt registered [13340.877244] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13341.203481] Key type .llcrypt unregistered [13341.205314] Key type ._llcrypt unregistered [13373.424957] Key type ._llcrypt registered [13373.427813] Key type .llcrypt registered [13373.578586] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13373.871364] Key type .llcrypt unregistered [13373.873096] Key type ._llcrypt unregistered [13407.381423] Key type ._llcrypt registered [13407.383166] Key type .llcrypt registered [13407.472610] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13407.858766] Key type .llcrypt unregistered [13407.862077] Key type ._llcrypt unregistered [13442.112356] Key type ._llcrypt registered [13442.116299] Key type .llcrypt registered [13442.240759] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13442.699244] Key type .llcrypt unregistered [13442.700880] Key type ._llcrypt unregistered [13510.352435] Key type ._llcrypt registered [13510.355282] Key type .llcrypt registered [13510.506985] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13510.895294] Key type .llcrypt unregistered [13510.898739] Key type ._llcrypt unregistered [13542.959233] Key type ._llcrypt registered [13542.960974] Key type .llcrypt registered [13543.057944] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13543.531523] Key type .llcrypt unregistered [13543.535658] Key type ._llcrypt unregistered [13574.734328] Key type ._llcrypt registered [13574.737802] Key type .llcrypt registered [13574.819683] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13575.106351] Key type .llcrypt unregistered [13575.107791] Key type ._llcrypt unregistered [13627.650681] Key type ._llcrypt registered [13627.652454] Key type .llcrypt registered [13627.790328] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [13645.040901] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13646.569381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13646.594040] alg: No test for adler32 (adler32-zlib) [13647.662205] Lustre: Lustre: Build Version: 2.17.56_50_g3eb67fe [13647.918540] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13649.616223] Key type lgssc registered [13650.641827] Lustre: Echo OBD driver; http://www.lustre.org/ [13687.373929] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13687.412653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13688.865488] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13688.922544] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13689.089712] Lustre: tst250fs-MDT0000: new disk, initializing [13689.261349] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13689.295609] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13695.051363] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13711.223495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13711.660626] Lustre: 374396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13711.836278] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13711.848933] Lustre: Skipped 1 previous similar message [13712.095262] Lustre: tst250fs-MDT0001: new disk, initializing [13712.253912] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13712.369442] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13712.396332] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13719.987520] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13725.218369] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13731.828275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13732.122971] Lustre: tst250fs-OST0000: new disk, initializing [13732.129341] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13732.144430] Lustre: 376040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13732.286723] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13735.012574] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13735.022811] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13735.128432] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13739.275995] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13750.944518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13751.097695] Lustre: tst250fs-OST0001: new disk, initializing [13751.100525] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13751.105216] Lustre: 376912:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13751.228188] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13757.942700] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13758.647963] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13758.672341] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13758.835713] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13764.606474] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13767.383420] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13767.390846] Lustre: Skipped 1 previous similar message [13794.273933] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13794.291452] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13794.307560] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13797.849850] Lustre: server umount tst250fs-MDT0000 complete [13802.061716] LustreError: 374387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786989774 with bad export cookie 5880182994482389631 [13802.064796] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13802.073164] LustreError: 374387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13802.423514] Lustre: server umount tst250fs-MDT0001 complete [13816.561294] Lustre: server umount tst250fs-OST0000 complete [13820.384175] Lustre: 372620:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786989777/real 1786989777] req@ffff94f64e09fb80 x1873794427858432/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786989793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13820.430499] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13820.449362] Lustre: Skipped 3 previous similar messages [13821.148376] Lustre: server umount tst250fs-OST0001 complete [13841.869349] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [13852.614786] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13884.767574] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:04:15 (1786989855) [13886.752567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13906.574904] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [13918.799849] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13958.338789] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13641 sec ========= 14:05:28 (1786989928) [13960.527058] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:05:30 (1786989930) === [13964.492379] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:05:35 (1786989935) === [14007.214138] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [14010.392797] Key type lgssc unregistered [14010.652886] LNet: 386056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14010.672458] LNetError: 386056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14011.756305] LNet: Removed LNI 192.168.202.136@tcp [14013.000613] Key type .llcrypt unregistered [14013.003993] Key type ._llcrypt unregistered