[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 510280154 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2464MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003242] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007827] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011091] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.022647] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027665] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030164] *** VALIDATE tmpfs *** [ 0.032438] *** VALIDATE proc *** [ 0.033289] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036186] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039016] Speculative Store Bypass: Vulnerable [ 0.042011] debug: unmapping init [mem 0xffffffff88e59000-0xffffffff88e60fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045706] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048025] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053316] rcu: Hierarchical SRCU implementation. [ 0.055413] smp: Bringing up secondary CPUs ... [ 0.056560] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.064093] smp: Brought up 1 node, 4 CPUs [ 0.066012] smpboot: Max logical packages: 1 [ 0.067012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152055] node 0 deferred pages initialised in 83ms [ 0.154604] devtmpfs: initialized [ 0.155225] x86/mm: Memory block size: 128MB [ 0.158009] gcov: version magic: 0x41383552 [ 0.160184] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161063] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162237] pinctrl core: initialized pinctrl subsystem [ 0.163203] [ 0.163517] ************************************************************* [ 0.164015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165013] ** ** [ 0.166010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167013] ** ** [ 0.168016] ** This means that this kernel is built to expose internal ** [ 0.169013] ** IOMMU data structures, which may compromise security on ** [ 0.170016] ** your system. ** [ 0.171018] ** ** [ 0.172015] ** If you see this message and you are not debugging the ** [ 0.173016] ** kernel, report this immediately to your vendor! ** [ 0.174016] ** ** [ 0.175016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176014] ************************************************************* [ 0.177692] NET: Registered protocol family 16 [ 0.178497] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181632] cpuidle: using governor menu [ 0.183894] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186500] PCI: Using configuration type 1 for base access [ 0.188127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.200045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.203200] cryptd: max_cpu_qlen set to 1000 [ 0.208229] ACPI: Added _OSI(Module Device) [ 0.209014] ACPI: Added _OSI(Processor Device) [ 0.211010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.212014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218411] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224624] ACPI: Interpreter enabled [ 0.225066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226012] ACPI: Using IOAPIC for interrupt routing [ 0.227132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228329] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237468] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.242374] acpiphp: Slot [2] registered [ 0.243116] acpiphp: Slot [5] registered [ 0.244110] acpiphp: Slot [6] registered [ 0.245129] acpiphp: Slot [7] registered [ 0.246112] acpiphp: Slot [8] registered [ 0.247127] acpiphp: Slot [9] registered [ 0.248097] acpiphp: Slot [10] registered [ 0.249124] acpiphp: Slot [3] registered [ 0.250109] acpiphp: Slot [4] registered [ 0.251087] acpiphp: Slot [11] registered [ 0.252084] acpiphp: Slot [12] registered [ 0.253091] acpiphp: Slot [13] registered [ 0.254101] acpiphp: Slot [14] registered [ 0.255111] acpiphp: Slot [15] registered [ 0.256088] acpiphp: Slot [16] registered [ 0.257085] acpiphp: Slot [17] registered [ 0.258000] acpiphp: Slot [18] registered [ 0.258000] acpiphp: Slot [19] registered [ 0.260134] acpiphp: Slot [20] registered [ 0.262110] acpiphp: Slot [21] registered [ 0.263109] acpiphp: Slot [22] registered [ 0.265110] acpiphp: Slot [23] registered [ 0.266133] acpiphp: Slot [24] registered [ 0.268118] acpiphp: Slot [25] registered [ 0.270102] acpiphp: Slot [26] registered [ 0.271148] acpiphp: Slot [27] registered [ 0.273110] acpiphp: Slot [28] registered [ 0.275148] acpiphp: Slot [29] registered [ 0.276104] acpiphp: Slot [30] registered [ 0.278119] acpiphp: Slot [31] registered [ 0.280149] PCI host bridge to bus 0000:00 [ 0.281022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.284024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.286029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.289027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.292052] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.295027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297306] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301187] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.304418] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.318016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.323065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.326020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.328355] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.332022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.334519] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338147] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.340064] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.343887] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.362093] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.368017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373828] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.388020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.408024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.417744] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.440019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.474018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.490655] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.506020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.522013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.560022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.570000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.584019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.602026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.650019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.668613] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.682014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.696015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.730017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.740118] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.750018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.760019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.779017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.789416] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.790293] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.795431] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.797385] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.799183] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.804223] iommu: Default domain type: Passthrough [ 0.806445] SCSI subsystem initialized [ 0.807095] ACPI: bus type USB registered [ 0.808104] usbcore: registered new interface driver usbfs [ 0.810057] usbcore: registered new interface driver hub [ 0.811058] usbcore: registered new device driver usb [ 0.812118] pps_core: LinuxPPS API ver. 1 registered [ 0.813007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.815070] PTP clock support registered [ 0.817100] EDAC MC: Ver: 3.0.0 [ 0.818163] PCI: Using ACPI for IRQ routing [ 0.820897] NetLabel: Initializing [ 0.822012] NetLabel: domain hash size = 128 [ 0.823008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.825088] NetLabel: unlabeled traffic allowed by default [ 0.827140] vgaarb: loaded [ 0.828190] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.829010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.833248] clocksource: Switched to clocksource kvm-clock [ 0.922109] VFS: Disk quotas dquot_6.6.0 [ 0.923392] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.925207] *** VALIDATE ramfs *** [ 0.925960] *** VALIDATE hugetlbfs *** [ 0.927593] pnp: PnP ACPI init [ 0.929398] pnp: PnP ACPI: found 6 devices [ 0.945596] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.947606] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.949683] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.951934] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.954474] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.956922] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.959847] NET: Registered protocol family 2 [ 0.962280] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.967186] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.969319] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.972819] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.975090] TCP: Hash tables configured (established 65536 bind 65536) [ 0.977019] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.979244] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.981243] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.983199] NET: Registered protocol family 1 [ 0.984937] RPC: Registered named UNIX socket transport module. [ 0.986250] RPC: Registered udp transport module. [ 0.987274] RPC: Registered tcp transport module. [ 0.988308] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.989799] NET: Registered protocol family 44 [ 0.990827] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.992216] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.993509] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.994879] PCI: CLS 0 bytes, default 64 [ 0.995884] Unpacking initramfs... [ 2.343513] debug: unmapping init [mem 0xffff8ee3bcc54000-0xffff8ee3bffbffff] [ 2.346980] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.348791] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.351169] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.871420] Initialise system trusted keyrings [ 2.873120] Key type blacklist registered [ 2.875024] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.884153] zbud: loaded [ 2.887145] *** VALIDATE nfs *** [ 2.888399] *** VALIDATE nfs4 *** [ 2.890018] pstore: using deflate compression [ 2.893377] Platform Keyring initialized [ 2.994730] NET: Registered protocol family 38 [ 2.996652] Key type asymmetric registered [ 2.998545] Asymmetric key parser 'x509' registered [ 3.000372] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.004623] io scheduler mq-deadline registered [ 3.006224] io scheduler kyber registered [ 3.007612] io scheduler bfq registered [ 3.009356] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.011974] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.014422] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.016823] ACPI: Power Button [PWRF] [ 3.021177] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.027646] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.036728] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.041717] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.053121] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.082967] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.113568] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.117178] Non-volatile memory driver v1.3 [ 3.118203] Linux agpgart interface v0.103 [ 3.141920] virtio_blk virtio1: [vda] 146000 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.143788] vda: detected capacity change from 0 to 74752000 [ 3.158806] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.160796] vdb: detected capacity change from 0 to 1073741824 [ 3.178029] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.180917] vdc: detected capacity change from 0 to 2621440000 [ 3.196047] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.198660] vdd: detected capacity change from 0 to 2621440000 [ 3.213156] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.215825] vde: detected capacity change from 0 to 4294967296 [ 3.238925] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.241510] vdf: detected capacity change from 0 to 4294967296 [ 3.252578] libphy: Fixed MDIO Bus: probed [ 3.256827] usbcore: registered new interface driver usbserial_generic [ 3.258377] usbserial: USB Serial support registered for generic [ 3.260467] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.263056] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.264208] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.265694] mousedev: PS/2 mouse device common for all mice [ 3.267338] rtc_cmos 00:05: RTC can wake from S4 [ 3.269382] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.270085] rtc_cmos 00:05: registered as rtc0 [ 3.274078] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.274402] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.276771] intel_pstate: CPU model not supported [ 3.279907] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.284219] hid: raw HID events driver (C) Jiri Kosina [ 3.286434] usbcore: registered new interface driver usbhid [ 3.288565] usbhid: USB HID core driver [ 3.290133] drop_monitor: Initializing network drop monitor service [ 3.292638] Initializing XFRM netlink socket [ 3.294638] NET: Registered protocol family 10 [ 3.297724] Segment Routing with IPv6 [ 3.299066] NET: Registered protocol family 17 [ 3.300529] mpls_gso: MPLS GSO support [ 3.306324] RAS: Correctable Errors collector initialized. [ 3.307591] AVX version of gcm_enc/dec engaged. [ 3.308861] AES CTR mode by8 optimization enabled [ 3.383667] sched_clock: Marking stable (3383647072, 0)->(4352790041, -969142969) [ 3.387784] registered taskstats version 1 [ 3.390058] Loading compiled-in X.509 certificates [ 3.391630] zswap: loaded using pool lzo/zbud [ 3.415388] Key type big_key registered [ 3.427666] Key type encrypted registered [ 3.429059] ima: No TPM chip found, activating TPM-bypass! [ 3.430979] ima: Allocated hash algorithm: sha1 [ 3.432543] ima: No architecture policies found [ 3.434131] evm: Initialising EVM extended attributes: [ 3.435857] evm: security.selinux [ 3.437116] evm: security.ima [ 3.438192] evm: security.capability [ 3.439484] evm: HMAC attrs: 0x1 [ 3.442168] rtc_cmos 00:05: setting system clock to 2026-08-19 08:33:30 UTC (1787128410) [ 3.447611] debug: unmapping init [mem 0xffffffff89e03000-0xffffffff89ffffff] [ 3.450367] debug: unmapping init [mem 0xffffffff88b82000-0xffffffff88e58fff] [ 3.463685] Write protecting the kernel read-only data: 28672k [ 3.466450] debug: unmapping init [mem 0xffffffff87203000-0xffffffff873fffff] [ 3.469176] debug: unmapping init [mem 0xffffffff87b14000-0xffffffff87bfffff] [ 3.502880] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.510981] systemd[1]: Detected virtualization kvm. [ 3.512749] systemd[1]: Detected architecture x86-64. [ 3.514500] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.538349] systemd[1]: No hostname configured. [ 3.540054] systemd[1]: Set hostname to . [ 3.542085] random: systemd: uninitialized urandom read (16 bytes read) [ 3.544477] systemd[1]: Initializing machine ID from random generator. [ 3.585436] random: ln: uninitialized urandom read (6 bytes read) [ 3.664063] random: systemd: uninitialized urandom read (16 bytes read) [ 3.666486] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.670147] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.674902] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.223164] device-mapper: uevent: version 1.0.3 [ 4.225158] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 4.678257] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 4.809562] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ 4.844918] scsi host0: ata_piix [ OK ] Started Hardware RNG Entropy Gatherer Daemon[ 4.850098] scsi host1: ata_piix . [ 4.853297] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.856219] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.211226] dracut-initqueue[593]: RTNETLINK answers: File exists [ 9.945283] random: crng init done [ 9.946401] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.308185] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.506727] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.762711] SELinux: Disabled at runtime. [ 11.817536] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.826336] systemd[1]: Detected virtualization kvm. [ 11.828213] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.329314] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.334146] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.339105] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.344294] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.347830] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.357256] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.367278] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ 12.516584] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.879859] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.216653] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.264495] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.392336] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.406470] EDAC sbridge: Ver: 1.1.2 [ 15.194687] Key type dns_resolver registered [ 15.496276] NFS: Registering the id_resolver key type [ 15.498607] Key type id_resolver registered [ 15.500479] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg136-server login: [ 106.754614] libcfs: loading out-of-tree module taints kernel. [ 106.803593] Key type ._llcrypt registered [ 106.806449] Key type .llcrypt registered [ 106.930371] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 129.870810] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 131.930361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.958542] alg: No test for adler32 (adler32-zlib) [ 133.547179] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 134.746490] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 136.600584] Key type lgssc registered [ 138.896318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 159.172351] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.807760] hrtimer: interrupt took 6268032 ns [ 200.323523] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 216.168820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 216.215352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.743681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 217.792916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 217.932090] Lustre: lustre-MDT0000: new disk, initializing [ 218.021098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.039968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 222.277558] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.784948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.932722] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 238.975813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 238.984587] Lustre: Skipped 1 previous similar message [ 239.058938] Lustre: lustre-MDT0001: new disk, initializing [ 239.171350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.223522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 239.251123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 244.990853] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.534043] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 261.510319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.814662] Lustre: lustre-OST0000: new disk, initializing [ 261.818322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 261.824249] Lustre: 8457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.898487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 269.347648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 269.366501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 269.471622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 269.603910] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 285.196953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 285.361298] Lustre: lustre-OST0001: new disk, initializing [ 285.365943] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 285.374132] Lustre: 9533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 285.459213] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 292.648556] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 294.497554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 294.504662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 294.589712] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 305.033509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 312.549571] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 319.477460] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.692750] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing yml_node [ 329.952462] Lustre: DEBUG MARKER: Client: 2.17.56.3 [ 332.247870] Lustre: DEBUG MARKER: MDS: 2.17.56.3 [ 334.660559] Lustre: DEBUG MARKER: OSS: 2.17.56.3 [ 336.295182] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 04:39:01 EDT 2026 [ 354.684381] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 356.507792] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 376.814280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.822110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.824778] Lustre: Skipped 3 previous similar messages [ 376.832642] Lustre: Skipped 3 previous similar messages [ 379.278911] Lustre: server umount lustre-MDT0000 complete [ 383.148153] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128790 with bad export cookie 15797637181079599854 [ 383.148958] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.167960] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 387.041048] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.042917] LustreError: 10360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.057825] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 387.080083] LustreError: 10360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 387.086734] Lustre: Skipped 1 previous similar message [ 388.066717] LustreError: 10360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.067328] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 388.096504] LustreError: 10360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 388.110499] Lustre: Skipped 1 previous similar message [ 389.689821] Lustre: server umount lustre-MDT0001 complete [ 393.960674] Lustre: server umount lustre-OST0000 complete [ 397.714381] Lustre: server umount lustre-OST0001 complete [ 405.033945] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 414.246915] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 448.681786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.936761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 448.956243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 449.036737] Lustre: lustre-MDT0000: new disk, initializing [ 449.106908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.129550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 454.073902] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 465.653147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.773167] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 465.864879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 465.869034] Lustre: Skipped 1 previous similar message [ 465.984870] Lustre: lustre-MDT0001: new disk, initializing [ 466.100467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.145844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.158988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 471.792462] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 481.567237] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 487.474574] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.279793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.654905] Lustre: lustre-OST0000: new disk, initializing [ 494.658636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 494.667895] Lustre: 17056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 494.822349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.197940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 496.210092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.343432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 501.614742] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 511.806525] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 517.028858] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 517.263323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 522.171631] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 522.422589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 527.330167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.334387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.345742] Lustre: Skipped 1 previous similar message [ 527.351535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 530.868504] Lustre: server umount lustre-OST0000 complete [ 537.571557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.576588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.591862] Lustre: Skipped 2 previous similar messages [ 537.617053] Lustre: Skipped 2 previous similar messages [ 541.375788] Lustre: server umount lustre-MDT0000 complete [ 542.699680] LustreError: 15096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.728701] LustreError: 15096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 545.893870] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128952 with bad export cookie 15797637181079601387 [ 545.899274] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.908094] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 546.239759] Lustre: server umount lustre-MDT0001 complete [ 553.656391] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:42:38 (1787128958) [ 575.691128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.014547] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.085748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 581.101701] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.436964] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 591.333648] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.349161] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 593.412918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.689803] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 608.019263] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 614.259537] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.365582] Lustre: Failing over lustre-MDT0000 [ 616.824237] Lustre: server umount lustre-MDT0000 complete [ 619.430608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.444355] Lustre: Skipped 1 previous similar message [ 619.450211] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.488519] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 635.616172] Lustre: 3652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129026/real 1787129026] req@ffff8ee427915180 x1873940097171456/t0(0) o400->MGC192.168.201.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787129042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 635.646126] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.350799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.922985] LustreError: 3651:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ee409225180 x1873940097174400/t0(0) o250->MGC192.168.201.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 646.383253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.395464] Lustre: Skipped 1 previous similar message [ 646.479839] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 651.748641] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 651.767062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 651.806915] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 652.243600] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 662.477064] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 667.108985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 667.117348] LustreError: Skipped 1 previous similar message [ 667.125762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.141338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 673.343356] Lustre: server umount lustre-MDT0000 complete [ 674.275039] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.309268] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 676.863756] LustreError: 21500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129083 with bad export cookie 15797637181079602801 [ 676.866881] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 676.885588] LustreError: 21500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 677.337776] Lustre: server umount lustre-MDT0001 complete [ 687.116236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 692.955673] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 705.589188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.795961] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 720.010724] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 726.080436] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 728.808141] Lustre: Failing over lustre-MDT0000 [ 729.286537] Lustre: server umount lustre-MDT0000 complete [ 731.623101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 731.636699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.665425] Lustre: Skipped 1 previous similar message [ 738.275622] LustreError: 23273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 738.317201] LustreError: 23273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 748.532663] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129139/real 1787129139] req@ffff8ee3037ef800 x1873940097202432/t0(0) o400->MGC192.168.201.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787129155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 748.564107] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 751.824114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.760785] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdb3c7b5114c0ca1b [ 758.794608] Lustre: MGC192.168.201.136@tcp: Connection restored to 0@lo (at 0@lo) [ 758.816694] Lustre: Skipped 1 previous similar message [ 759.307659] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 764.406136] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 764.435870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 764.461219] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 765.052561] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 776.388206] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 784.865562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 784.868610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.874143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 784.874152] Lustre: Skipped 2 previous similar messages [ 784.919827] Lustre: Skipped 2 previous similar messages [ 795.105478] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 797.308918] Lustre: server umount lustre-MDT0000 complete [ 801.628753] LustreError: 23258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129208 with bad export cookie 15797637181079603739 [ 801.632722] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 801.649258] LustreError: 23258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 802.206937] Lustre: server umount lustre-MDT0001 complete [ 810.725941] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:46:55 (1787129215) [ 829.828922] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 840.713659] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 885.950713] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 900.922089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 901.237467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 901.376568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 901.518812] Lustre: lustre-MDT0000: new disk, initializing [ 901.661333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.672314] Lustre: Skipped 3 previous similar messages [ 901.703272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 907.308495] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 920.122773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 920.211378] Lustre: 29658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 920.388615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 925.492765] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 935.715597] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 942.249286] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 949.496604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 949.795632] Lustre: 31629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 951.251839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 951.322503] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 951.421872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 951.439456] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 956.978510] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 967.722467] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 974.420769] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 997.346564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 997.357836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.377918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 997.384379] Lustre: Skipped 5 previous similar messages [ 1011.168526] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1011.401389] Lustre: server umount lustre-OST0000 complete [ 1022.297690] Lustre: server umount lustre-MDT0000 complete [ 1026.535108] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.556243] LustreError: 29672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 1027.078979] LustreError: 29653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129434 with bad export cookie 15797637181079604152 [ 1027.081085] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.101279] LustreError: 29653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1027.694198] Lustre: server umount lustre-MDT0001 complete [ 1040.707708] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1044.506795] Key type lgssc unregistered [ 1045.002786] LNet: 33615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.026350] LNetError: 33615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.052333] LNet: Removed LNI 192.168.201.136@tcp [ 1046.448232] Key type .llcrypt unregistered [ 1046.454919] Key type ._llcrypt unregistered [ 1078.216188] Key type ._llcrypt registered [ 1078.218756] Key type .llcrypt registered [ 1078.367233] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1080.197101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1080.355046] alg: No test for adler32 (adler32-zlib) [ 1081.604225] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1081.927570] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1083.616190] Key type lgssc registered [ 1084.947950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1094.527536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1094.560224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.278578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.682699] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1106.404526] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.449452] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1111.547524] LustreError: 35030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.859678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.130977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1121.577210] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1135.154682] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1142.119271] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1150.728987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1151.243663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.531778] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1160.651964] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1166.710087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1171.313773] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1189.600726] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:53:14 (1787129594) [ 1192.037872] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1194.513742] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1203.564457] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:53:28 (1787129608) [ 1214.413087] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:53:39 (1787129619) [ 1215.476408] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1224.844345] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1224.856425] Lustre: Skipped 2 previous similar messages [ 1226.794641] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:53:51 (1787129631) [ 1229.012387] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1229.021356] Lustre: Skipped 2 previous similar messages [ 1239.406649] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1239.413637] Lustre: Skipped 4 previous similar messages [ 1257.972090] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:54:22 (1787129662) [ 1261.604343] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1265.471218] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1275.700836] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:54:40 (1787129680) [ 1278.395530] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1278.406097] Lustre: Skipped 6 previous similar messages [ 1295.387854] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:55:00 (1787129700) [ 1315.302290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1315.305523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.311690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1315.329689] Lustre: Skipped 1 previous similar message [ 1315.335698] Lustre: Skipped 1 previous similar message [ 1320.475866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1320.483267] Lustre: Skipped 1 previous similar message [ 1325.539117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.546351] Lustre: Skipped 1 previous similar message [ 1329.120276] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1329.259986] Lustre: server umount lustre-OST0000 complete [ 1335.265765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1335.276538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.288534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1340.354046] Lustre: server umount lustre-MDT0000 complete [ 1340.898996] LustreError: 35758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.929349] LustreError: 35758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1344.879115] LustreError: 35767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129751 with bad export cookie 13466849550658549894 [ 1344.879757] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1344.889842] LustreError: 35767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1345.419721] Lustre: server umount lustre-MDT0001 complete [ 1357.997939] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1361.257653] Key type lgssc unregistered [ 1361.591809] LNet: 42898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.606158] LNetError: 42898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1361.621488] LNet: Removed LNI 192.168.201.136@tcp [ 1362.653816] Key type .llcrypt unregistered [ 1362.656502] Key type ._llcrypt unregistered [ 1385.322934] Key type ._llcrypt registered [ 1385.325325] Key type .llcrypt registered [ 1385.564951] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 1403.822358] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1405.277860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1405.335675] alg: No test for adler32 (adler32-zlib) [ 1406.379470] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1406.610469] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1408.272451] Key type lgssc registered [ 1409.970157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1456.264701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1456.312614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.759658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1457.835717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1457.933687] Lustre: lustre-MDT0000: new disk, initializing [ 1458.053273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.085065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1463.922885] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1476.972759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.177971] Lustre: 46057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1477.279952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1477.288669] Lustre: Skipped 1 previous similar message [ 1477.412767] Lustre: lustre-MDT0001: new disk, initializing [ 1477.523779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.565800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1477.582703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1483.457323] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1496.903363] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1503.474723] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1513.224756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1513.622171] Lustre: lustre-OST0000: new disk, initializing [ 1513.629061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1513.660619] Lustre: 48024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1513.851278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1519.685972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1519.700065] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1519.912063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1522.654954] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1534.994422] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1541.845189] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1542.189280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1548.229384] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1548.639341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1555.938963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1555.949325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.965066] LustreError: Skipped 1 previous similar message [ 1555.965964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1555.971911] Lustre: Skipped 1 previous similar message [ 1556.026653] Lustre: Skipped 1 previous similar message [ 1557.488080] Lustre: server umount lustre-OST0000 complete [ 1564.646090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1564.662370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.680847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.177810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.178904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.211567] Lustre: Skipped 1 previous similar message [ 1568.629250] Lustre: server umount lustre-MDT0000 complete [ 1573.625423] LustreError: 46049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129980 with bad export cookie 11291288798949298716 [ 1573.632524] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1573.641542] LustreError: 46049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1574.297658] Lustre: server umount lustre-MDT0001 complete [ 1586.903792] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:59:52 (1787129992) [ 1594.981335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.708858] LustreError: 50487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.752637] LustreError: 50487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1595.850148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.000459] LustreError: 50488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.669371] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1605.092988] LustreError: 50487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1610.234418] LustreError: 50488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1612.139982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1612.378805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1617.814507] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1625.884725] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1632.786863] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1641.036074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1641.366251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.933876] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1661.604658] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1670.923929] LustreError: 53034:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1672.027223] Lustre: Modifying parameter general.timeout=1 in log params [ 1673.123624] Lustre: Modifying parameter general.timeout=2 in log params [ 1675.867728] Lustre: Modifying parameter general.timeout=5 in log params [ 1675.875723] Lustre: Skipped 2 previous similar messages [ 1680.899500] Lustre: Modifying parameter general.timeout=9 in log params [ 1680.904707] Lustre: Skipped 3 previous similar messages [ 1689.180979] Lustre: Modifying parameter general.timeout=16 in log params [ 1689.183082] Lustre: Skipped 6 previous similar messages [ 1705.660463] Lustre: Modifying parameter general.timeout=31 in log params [ 1705.668256] Lustre: Skipped 14 previous similar messages [ 1737.732771] Lustre: Modifying parameter general.timeout=61 in log params [ 1737.736532] Lustre: Skipped 29 previous similar messages [ 1801.952993] Lustre: Modifying parameter general.timeout=121 in log params [ 1801.957512] Lustre: Skipped 59 previous similar messages [ 1823.254776] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:03:47 (1787130227) [ 1826.280874] Lustre: 59986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1828.185922] Lustre: 60085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1830.357295] Lustre: 60183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1835.911527] Lustre: 60497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1835.932277] Lustre: 60497:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1865.670443] Lustre: 61859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1865.687178] LustreError: 61859:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1871.321703] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:04:35 (1787130275) [ 1888.739598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1888.764406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1888.781744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.268316] Lustre: server umount lustre-MDT0000 complete [ 1894.882442] LustreError: 50488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.910410] LustreError: 50488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1896.371217] LustreError: 52333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130303 with bad export cookie 11291288798949299598 [ 1896.388964] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1897.001191] Lustre: server umount lustre-MDT0001 complete [ 1912.038990] Lustre: server umount lustre-OST0000 complete [ 1941.580303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.260710] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1942.324757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1942.338386] Lustre: Skipped 31 previous similar messages [ 1942.602731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.807808] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1959.531444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.709616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1959.782707] Lustre: 64394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1966.004532] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1976.254452] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1982.599480] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1990.810661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1990.978687] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1991.338582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.347822] Lustre: Skipped 1 previous similar message [ 1998.757835] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2006.589019] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2008.362140] Lustre: 66962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2015.713818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2015.733075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.751956] Lustre: Skipped 2 previous similar messages [ 2015.757331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2015.765499] Lustre: Skipped 4 previous similar messages [ 2020.846437] LustreError: 66240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.868242] LustreError: 66240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2021.028647] Lustre: server umount lustre-OST0000 complete [ 2025.955393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.962505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.975927] Lustre: Skipped 2 previous similar messages [ 2039.264123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2039.454837] Lustre: server umount lustre-MDT0000 complete [ 2041.321070] LustreError: 64404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.347161] LustreError: 64404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2043.637039] LustreError: 64384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130450 with bad export cookie 11291288798949302804 [ 2043.640292] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2043.645292] LustreError: 64384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2043.978818] Lustre: server umount lustre-MDT0001 complete [ 2056.137533] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2059.147538] Key type lgssc unregistered [ 2059.611795] LNet: 68171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.624831] LNetError: 68171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.686121] LNet: Removed LNI 192.168.201.136@tcp [ 2060.646356] Key type .llcrypt unregistered [ 2060.648770] Key type ._llcrypt unregistered [ 2086.056857] Key type ._llcrypt registered [ 2086.058845] Key type .llcrypt registered [ 2086.153030] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2087.326312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2087.402408] alg: No test for adler32 (adler32-zlib) [ 2088.521554] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2088.861996] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2090.601434] Key type lgssc registered [ 2091.959512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2099.941528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2099.955689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.398674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2105.865973] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2110.947777] LustreError: 69587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.970774] LustreError: 69587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2114.878702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.517818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2120.509071] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2128.619916] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2135.113190] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2143.154420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.004821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.700831] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2160.091495] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2169.375110] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2182.113664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2182.120880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.132282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2186.221949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2186.223023] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.231820] Lustre: Skipped 1 previous similar message [ 2186.257814] Lustre: Skipped 1 previous similar message [ 2187.668329] Lustre: server umount lustre-MDT0000 complete [ 2191.354562] LustreError: 72631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.382467] LustreError: 72631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2192.792806] LustreError: 69569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130599 with bad export cookie 3982095005904555064 [ 2192.808890] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2192.818944] LustreError: 69569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2193.277337] Lustre: server umount lustre-MDT0001 complete [ 2209.455635] Lustre: server umount lustre-OST0000 complete [ 2237.422668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.749599] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2237.800469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2238.041157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.619248] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2252.910542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.078937] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2253.147560] Lustre: 74461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2253.218712] LustreError: 74471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.149493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2259.149621] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2266.267705] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2271.209361] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2277.968683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2278.282256] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2278.752728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.389829] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2292.779813] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2294.609703] Lustre: 77037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2296.333114] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2301.412742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2301.418578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.451634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2301.457449] Lustre: Skipped 1 previous similar message [ 2306.537379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.548301] Lustre: Skipped 1 previous similar message [ 2307.622435] Lustre: server umount lustre-OST0000 complete [ 2311.650043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.655415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.663812] Lustre: Skipped 2 previous similar messages [ 2317.779082] Lustre: server umount lustre-MDT0000 complete [ 2321.889225] LustreError: 74472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.921053] LustreError: 74472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2322.149146] LustreError: 77040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130729 with bad export cookie 3982095005904555911 [ 2322.172491] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.180244] LustreError: 77040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2322.575545] Lustre: server umount lustre-MDT0001 complete [ 2339.539955] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2342.956828] Key type lgssc unregistered [ 2343.576660] LNet: 78248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.582693] LNetError: 78248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.609423] LNet: Removed LNI 192.168.201.136@tcp [ 2344.580377] Key type .llcrypt unregistered [ 2344.582620] Key type ._llcrypt unregistered [ 2371.746888] Key type ._llcrypt registered [ 2371.750687] Key type .llcrypt registered [ 2371.955171] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2373.157272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2373.236536] alg: No test for adler32 (adler32-zlib) [ 2374.476520] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2374.829468] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2376.529848] Key type lgssc registered [ 2378.111901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2386.954492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2386.967952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2388.522752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.275253] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2398.689231] LustreError: 79663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.701558] LustreError: 79663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2402.558587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.192895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2408.911230] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2416.895751] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2422.566349] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2430.191552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.649430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.511301] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2446.165404] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2452.882776] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2453.796459] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2455.893875] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2455.897965] Lustre: Skipped 1 previous similar message [ 2457.902756] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2457.905618] Lustre: Skipped 1 previous similar message [ 2462.059697] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2462.068445] Lustre: Skipped 3 previous similar messages [ 2470.349190] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2470.359066] Lustre: Skipped 8 previous similar messages [ 2486.914052] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2486.920272] Lustre: Skipped 15 previous similar messages [ 2519.612755] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2519.617291] Lustre: Skipped 33 previous similar messages [ 2560.506331] Lustre: 87767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2562.868487] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:16:08 (1787130968) [ 2564.707634] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2566.847486] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:16:12 (1787130972) [ 2572.258351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2572.267794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.276710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2575.336760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2575.340691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2575.343331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.356811] Lustre: Skipped 1 previous similar message [ 2577.070882] Lustre: server umount lustre-OST0000 complete [ 2585.584339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.615146] Lustre: Skipped 1 previous similar message [ 2585.615200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2585.640874] Lustre: Skipped 1 previous similar message [ 2589.971056] Lustre: server umount lustre-MDT0000 complete [ 2590.691472] LustreError: 79664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.736214] LustreError: 79664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2595.814623] LustreError: 81671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.827984] LustreError: 81671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2596.015375] LustreError: 79644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131003 with bad export cookie 7584347722831102321 [ 2596.017709] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2596.031959] LustreError: 79644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2596.327490] Lustre: server umount lustre-MDT0001 complete [ 2607.467788] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2609.494604] Key type lgssc unregistered [ 2609.690724] LNet: 89018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2609.699594] LNetError: 89018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2609.716241] LNet: Removed LNI 192.168.201.136@tcp [ 2610.367296] Key type .llcrypt unregistered [ 2610.369781] Key type ._llcrypt unregistered [ 2614.529475] Key type ._llcrypt registered [ 2614.532159] Key type .llcrypt registered [ 2614.646853] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2619.178442] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules [ 2620.134055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2620.256480] alg: No test for adler32 (adler32-zlib) [ 2621.504826] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2621.768261] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2623.456180] Key type lgssc registered [ 2624.707461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2626.020413] LustreError: 90229:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2645.194883] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2648.237753] LustreError: 90229:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2655.435291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2655.487569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.018819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.086475] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2666.467879] LustreError: 91249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.490621] LustreError: 91249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2671.594848] LustreError: 91250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.993914] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:17:57 (1787131077) [ 2676.721607] LustreError: 91249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.200502] LustreError: 91281:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2677.212842] LustreError: 91281:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2677.220763] LustreError: 91281:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2677.624466] Lustre: server umount lustre-MDT0000 complete [ 2692.599323] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2695.436920] Key type lgssc unregistered [ 2695.781784] LNet: 92775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2695.789575] LNetError: 92775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2695.801212] LNet: Removed LNI 192.168.201.136@tcp [ 2696.787465] Key type .llcrypt unregistered [ 2696.792538] Key type ._llcrypt unregistered [ 2714.456086] Key type ._llcrypt registered [ 2714.457328] Key type .llcrypt registered [ 2714.600150] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2715.624319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2715.731593] alg: No test for adler32 (adler32-zlib) [ 2716.911706] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2717.331418] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2719.136811] Key type lgssc registered [ 2720.320670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2727.704926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2727.722387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2729.494915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.586427] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2739.697702] LustreError: 93995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.731952] LustreError: 93995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2743.993818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.487872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.017513] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2757.472417] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2763.557185] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2770.771696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2771.424715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.844513] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2785.713459] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2841.301669] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:20:46 (1787131246) [ 2843.526540] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2846.158500] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:20:50 (1787131250) [ 2851.302526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.314380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.327695] Lustre: Skipped 1 previous similar message [ 2851.352466] Lustre: Skipped 2 previous similar messages [ 2856.426351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.419591] Lustre: server umount lustre-MDT0000 complete [ 2861.539860] LustreError: 93304:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000065815f08 x1873942805531904/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2861.540463] LustreError: 93976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131268 with bad export cookie 15271625079764139975 [ 2861.542482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.542490] Lustre: Skipped 1 previous similar message [ 2861.584055] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2861.584935] LustreError: 93990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.593785] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2861.593793] Lustre: Skipped 2 previous similar messages [ 2861.599344] LustreError: 93976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2861.978423] Lustre: server umount lustre-MDT0001 complete [ 2869.120165] Lustre: server umount lustre-OST0000 complete [ 2882.401744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.936808] LustreError: 98300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.167167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.162757] LustreError: 98300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.187791] LustreError: 98300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2888.287983] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2893.282726] LustreError: 98301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2897.687983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.936590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2904.359835] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2912.635869] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2918.929863] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2932.669492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2932.851748] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2932.859311] LustreError: 98287:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2932.900902] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2932.909983] LustreError: 100384:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2932.917607] LustreError: 100384:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2932.924044] LustreError: 100384:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2933.027802] Lustre: server umount lustre-OST0000 complete [ 2933.029532] LustreError: 100384:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2941.529627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.750518] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2941.764282] LustreError: 98289:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2941.814874] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2941.826938] LustreError: 100849:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2941.833253] LustreError: 100849:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2941.838942] LustreError: 100849:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2941.899432] Lustre: server umount lustre-OST0000 complete [ 2941.902393] LustreError: 100849:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2953.375479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2953.752266] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2953.760073] Lustre: Found index 0 for lustre-OST0000, updating log [ 2953.779104] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2953.899248] Lustre: lustre-OST0000: new disk, initializing [ 2953.903865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2953.926241] Lustre: 101384:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2954.172143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.361675] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2966.388957] Lustre: Failing over lustre-OST0000 [ 2966.521212] Lustre: server umount lustre-OST0000 complete [ 2970.847139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.859621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2975.205157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2975.210569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.226261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2977.215753] Lustre: server umount lustre-MDT0000 complete [ 2980.922074] LustreError: 98282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131387 with bad export cookie 15271625079764142726 [ 2980.930499] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2981.223314] Lustre: server umount lustre-MDT0001 complete [ 2990.368071] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:23:15 (1787131395) [ 3010.437616] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3021.369122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3021.707102] LustreError: 103729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3021.811396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.934988] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3035.362768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.559847] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3044.746358] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3052.202112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3062.447181] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3071.837483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3071.944953] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3072.062615] Lustre: lustre-OST0001: new disk, initializing [ 3072.065592] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3072.077449] Lustre: 106049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3072.954300] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3072.964967] Lustre: Skipped 2 previous similar messages [ 3078.313646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3078.329982] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3078.469337] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3080.833183] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3089.231622] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3094.461204] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3099.080214] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [ 3099.089436] Lustre: Skipped 2 previous similar messages [ 3099.107558] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.134142] Lustre: Skipped 2 previous similar messages [ 3104.915548] Lustre: server umount lustre-MDT0001 complete [ 3109.300588] LustreError: 103724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.328444] LustreError: 103724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3115.862609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.070228] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3116.073180] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3116.097735] Lustre: 103719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3117.191765] LustreError: 107576:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3117.212614] LustreError: 107576:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3117.229840] LustreError: 103721:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3117.386184] LustreError: 107589:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.136@tcp: cfg command failed: rc = -17 [ 3117.401345] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3117.401345] [ 3122.177033] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3123.543558] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3138.230544] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:25:43 (1787131543) [ 3156.451320] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3175.034648] Lustre: 109669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3199.070356] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3202.562500] Lustre: 110805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3295.713825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3295.726599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.740173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3295.752414] Lustre: Skipped 7 previous similar messages [ 3300.403856] Lustre: server umount lustre-MDT0000 complete [ 3301.346022] LustreError: 105244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.369180] LustreError: 105244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3304.467917] LustreError: 103709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131711 with bad export cookie 15271625079764143720 [ 3304.476857] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3304.485666] LustreError: 103709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3304.943445] Lustre: server umount lustre-MDT0001 complete [ 3319.823713] Lustre: server umount lustre-OST0000 complete [ 3322.593162] Lustre: 93307:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787131713/real 1787131713] req@ffff8ee433cb5500 x1873942805845888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787131729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3322.625616] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.638292] Lustre: Skipped 3 previous similar messages [ 3324.333165] Lustre: server umount lustre-OST0001 complete [ 3330.674633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3341.967925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3352.717251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3365.813156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3391.004424] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3402.886145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3402.896398] Lustre: lustre-MDT0000: reset Object Index mappings [ 3403.228097] LustreError: 114758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3403.344684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.349931] Lustre: Skipped 1 previous similar message [ 3408.079323] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3419.774635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3419.801581] Lustre: lustre-MDT0001: reset Object Index mappings [ 3427.295747] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3430.864626] Lustre: 115907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3438.665581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3445.387558] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3446.350456] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3448.443598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3456.624662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3462.660114] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3462.665663] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3465.504171] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3474.016644] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3478.043479] Lustre: 117783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3538.837458] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:32:23 (1787131943) [ 3559.922221] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3559.940878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3559.943799] Lustre: Skipped 4 previous similar messages [ 3566.457943] Lustre: server umount lustre-MDT0000 complete [ 3570.151483] LustreError: 118029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.172090] LustreError: 118029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3570.920677] LustreError: 114740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131977 with bad export cookie 15271625079764190634 [ 3570.921946] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3570.948038] LustreError: 114740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3577.493217] Lustre: server umount lustre-MDT0001 complete [ 3587.892836] Lustre: server umount lustre-OST0000 complete [ 3598.322931] Lustre: server umount lustre-OST0001 complete [ 3604.655612] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 3613.930800] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3652.949436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.163503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3653.185661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3653.252774] Lustre: lustre-MDT0000: new disk, initializing [ 3653.326027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.332721] Lustre: Skipped 3 previous similar messages [ 3653.344749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3657.861730] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3665.329828] Lustre: server umount lustre-MDT0000 complete [ 3676.036972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.392449] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3676.412476] Lustre: 122056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3676.437250] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3682.152619] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3687.735843] Lustre: server umount lustre-MDT0000 complete [ 3696.488626] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:35:01 (1787132101) [ 3698.693315] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3700.950688] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:35:05 (1787132105) [ 3723.912465] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3733.799316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.814117] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3746.288195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3746.446487] Lustre: 124107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3746.535155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3746.539477] Lustre: Skipped 1 previous similar message [ 3746.609927] Lustre: lustre-MDT0001: new disk, initializing [ 3746.760067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3746.773860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3751.166440] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3756.630324] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3764.001813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3764.451902] Lustre: lustre-OST0000: new disk, initializing [ 3764.455802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3764.468173] Lustre: 125685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3766.334642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3766.347571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3766.459654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3771.673469] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3783.006866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3783.149939] Lustre: lustre-OST0001: new disk, initializing [ 3783.156209] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3783.165948] Lustre: 126556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3784.576679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3784.595684] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3784.784404] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3790.279074] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3801.172983] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3805.482717] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4244.272850] Lustre: 132978:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4244.276828] Lustre: 132978:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4244.279904] Lustre: 132978:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4244.283656] Lustre: 132978:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4244.288341] Lustre: 132978:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4244.292781] Lustre: 132978:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4551.981455] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:49:16 (1787132956) [ 4568.035231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.041337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4568.051060] Lustre: Skipped 7 previous similar messages [ 4568.076517] Lustre: Skipped 12 previous similar messages [ 4573.854668] Lustre: server umount lustre-MDT0000 complete [ 4577.984951] LustreError: 124097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787132985 with bad export cookie 15271625079764258940 [ 4577.991643] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4577.996596] LustreError: 124097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4578.276177] LustreError: 127385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4578.280613] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.286438] LustreError: 127385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4578.290156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4578.290166] Lustre: Skipped 4 previous similar messages [ 4578.335499] Lustre: Skipped 2 previous similar messages [ 4584.710601] Lustre: server umount lustre-MDT0001 complete [ 4595.814820] Lustre: server umount lustre-OST0000 complete [ 4614.112532] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4614.501553] Lustre: server umount lustre-OST0001 complete [ 4623.690142] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4636.008684] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4677.661886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.938660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4677.994925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4678.161673] Lustre: lustre-MDT0000: new disk, initializing [ 4678.265813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.273985] Lustre: Skipped 5 previous similar messages [ 4678.309611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4683.231712] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4694.922135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4695.043317] Lustre: 136041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4695.159474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4695.169095] Lustre: Skipped 1 previous similar message [ 4695.306313] Lustre: lustre-MDT0001: new disk, initializing [ 4695.443948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4695.453506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4700.641228] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4712.634676] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4718.560270] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4725.942903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4726.268302] Lustre: lustre-OST0000: new disk, initializing [ 4726.272138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4726.281566] Lustre: 138012:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4726.427679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.434513] Lustre: Skipped 1 previous similar message [ 4727.496416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4727.513630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4727.683026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4732.912975] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4743.078380] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4754.680920] Lustre: lustre-MDD0000: changelog on [ 4759.236443] Lustre: lustre-MDD0001: changelog on [ 6584.994445] Lustre: lustre-MDD0001: changelog off [ 6587.011677] Lustre: 136047:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6588.258961] Lustre: lustre-MDD0000: changelog off [ 6596.580843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.588807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6596.588817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6596.588820] Lustre: Skipped 3 previous similar messages [ 6596.592943] Lustre: Skipped 1 previous similar message [ 6596.613464] LustreError: Skipped 1 previous similar message [ 6601.703308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6601.712523] Lustre: Skipped 2 previous similar messages [ 6606.821554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6606.833275] Lustre: Skipped 1 previous similar message [ 6609.376751] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6609.537773] Lustre: server umount lustre-OST0000 complete [ 6615.525681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6615.535506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6615.558315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6627.296304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6627.593854] Lustre: server umount lustre-MDT0000 complete [ 6631.509959] LustreError: 136844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135038 with bad export cookie 15271625079765080754 [ 6631.514262] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6631.519377] LustreError: 136844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6631.892975] Lustre: server umount lustre-MDT0001 complete [ 6642.958674] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 6645.529388] Key type lgssc unregistered [ 6645.805686] LNet: 147142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6645.809865] LNetError: 147142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6645.825129] LNet: Removed LNI 192.168.201.136@tcp [ 6646.669162] Key type .llcrypt unregistered [ 6646.670953] Key type ._llcrypt unregistered [ 6667.429111] Key type ._llcrypt registered [ 6667.434980] Key type .llcrypt registered [ 6667.560701] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 6686.037240] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6687.087285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6687.158146] alg: No test for adler32 (adler32-zlib) [ 6688.171852] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 6688.363114] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 6690.016182] Key type lgssc registered [ 6691.752282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6727.906128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6727.931487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.390202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6729.481340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6729.611357] Lustre: lustre-MDT0000: new disk, initializing [ 6729.716515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6729.738538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6734.150855] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6745.662338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6745.858209] Lustre: 150346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6746.032166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6746.042694] Lustre: Skipped 1 previous similar message [ 6746.116381] Lustre: lustre-MDT0001: new disk, initializing [ 6746.205879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6746.237718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6746.245838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6751.814737] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6762.656351] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6769.107738] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6777.131174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.476420] Lustre: lustre-OST0000: new disk, initializing [ 6777.481609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6777.493416] Lustre: 152314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6777.579354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.951491] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6787.220117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6787.230368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6787.451962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6794.586525] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6800.549419] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_module obdecho/obdecho [ 6802.073024] LustreError: 153318:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6802.081764] LustreError: 153318:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6803.687819] LustreError: 153414:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6817.761316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6817.772761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.778784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6818.277207] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.289634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6818.291094] Lustre: Skipped 1 previous similar message [ 6818.306143] Lustre: Skipped 2 previous similar messages [ 6821.560728] Lustre: server umount lustre-MDT0000 complete [ 6825.889961] LustreError: 150337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135232 with bad export cookie 4347961284441324322 [ 6825.895267] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6825.899610] LustreError: 150337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6826.365429] Lustre: server umount lustre-MDT0001 complete [ 6841.610063] Lustre: server umount lustre-OST0000 complete [ 6863.920260] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 6875.526776] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6912.381163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.577143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6912.603306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6912.670210] Lustre: lustre-MDT0000: new disk, initializing [ 6912.733163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6912.747496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6917.296557] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6928.803315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.985558] Lustre: 157130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6929.135545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6929.146357] Lustre: Skipped 1 previous similar message [ 6929.280684] Lustre: lustre-MDT0001: new disk, initializing [ 6929.434496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6929.504050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6929.531506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6934.476473] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6945.499798] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6952.285814] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6959.475353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6959.757526] Lustre: lustre-OST0000: new disk, initializing [ 6959.764537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6959.772946] Lustre: 159099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6959.852751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6966.400720] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6967.974966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6968.004969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6968.175520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6976.472947] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6982.297503] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6982.640634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6988.810074] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6989.173893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6993.893493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6993.896296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6993.923238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6993.931789] Lustre: Skipped 1 previous similar message [ 6997.879701] Lustre: server umount lustre-OST0000 complete [ 7004.132875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7004.150878] Lustre: Skipped 1 previous similar message [ 7004.157863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7004.165131] Lustre: Skipped 1 previous similar message [ 7008.391193] Lustre: server umount lustre-MDT0000 complete [ 7009.249734] LustreError: 157138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7009.268161] LustreError: 157138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7012.101883] LustreError: 159109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135419 with bad export cookie 4347961284441325281 [ 7012.112736] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7012.119302] LustreError: 159109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7012.699363] Lustre: server umount lustre-MDT0001 complete [ 7016.307079] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:30:21 (1787135421) [ 7018.019861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7020.100382] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:30:25 (1787135425) [ 7028.190358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7028.798813] LustreError: 161512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.850982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.363965] LustreError: 161513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.892916] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7038.501263] LustreError: 161512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.553769] LustreError: 161513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7045.037387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7050.186730] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7057.613735] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7063.186802] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7070.304415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7070.662193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.665950] Lustre: Skipped 1 previous similar message [ 7077.328403] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7086.453286] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.811202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7096.822081] LustreError: Skipped 1 previous similar message [ 7096.826154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.848497] Lustre: Skipped 1 previous similar message [ 7096.854659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7098.447559] Lustre: server umount lustre-MDT0000 complete [ 7100.392210] LustreError: 161507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7100.414436] LustreError: 161507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7101.715512] LustreError: 161492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135508 with bad export cookie 4347961284441326163 [ 7101.729189] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7105.521881] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7105.540040] Lustre: Skipped 2 previous similar messages [ 7105.550293] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7105.558047] Lustre: Skipped 2 previous similar messages [ 7116.256358] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7116.710916] Lustre: server umount lustre-MDT0001 complete [ 7123.333688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7135.907629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7136.146585] LustreError: 165354:0:(llog.c:1655:llog_backup()) MGC192.168.201.136@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7136.156737] Lustre: 165354:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.136@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7141.345529] LustreError: 165359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7141.368932] LustreError: 165359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7174.045626] LustreError: 165360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7174.068148] LustreError: 165360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 7196.856729] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7196.868742] Lustre: 165354:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.136@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7237.619333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7238.567222] LustreError: 165376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7238.607506] LustreError: 165376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 7242.729705] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7242.997160] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7252.458664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.195977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7271.394404] LustreError: 148567:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b290ae6b x1873946969511680/t0(0) o250->MGC192.168.201.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7277.079643] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7277.983766] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7287.697473] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7289.508493] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7295.152324] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7296.701344] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7317.986063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7317.998103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7317.998115] Lustre: Skipped 4 previous similar messages [ 7318.004639] Lustre: Skipped 2 previous similar messages [ 7328.229114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7328.476894] Lustre: server umount lustre-MDT0000 complete [ 7332.601110] LustreError: 161492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135739 with bad export cookie 4347961284441332652 [ 7332.606158] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7332.613969] LustreError: 161492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7332.938550] Lustre: server umount lustre-MDT0001 complete [ 7348.512682] Lustre: server umount lustre-OST0000 complete [ 7370.763570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 7381.193319] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7418.291171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7418.505458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7418.522862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7418.601507] Lustre: lustre-MDT0000: new disk, initializing [ 7418.667766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.671566] Lustre: Skipped 1 previous similar message [ 7418.678181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7423.420874] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7434.398046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7434.519088] Lustre: 170468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7434.870767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7439.499636] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7449.045912] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7455.186938] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7461.493387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7461.820727] Lustre: lustre-OST0000: new disk, initializing [ 7461.826297] Lustre: Skipped 1 previous similar message [ 7461.830903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7461.836148] Lustre: Skipped 2 previous similar messages [ 7461.845296] Lustre: 172437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7467.592768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7467.600373] Lustre: Skipped 1 previous similar message [ 7467.604112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7467.726400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7468.009105] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7477.096674] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7481.931833] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7482.135111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7486.976977] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7487.175540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7493.601470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7493.603592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7493.612346] LustreError: Skipped 1 previous similar message [ 7493.621243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7493.628202] Lustre: Skipped 1 previous similar message [ 7493.639272] Lustre: Skipped 9 previous similar messages [ 7495.420620] Lustre: server umount lustre-OST0000 complete [ 7505.911457] Lustre: server umount lustre-MDT0000 complete [ 7508.962831] LustreError: 171630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.980442] LustreError: 171630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [ 7509.341123] LustreError: 170458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135916 with bad export cookie 4347961284441349452 [ 7509.346296] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7509.350662] LustreError: 170458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7509.852836] Lustre: server umount lustre-MDT0001 complete [ 7518.231205] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:38:43 (1787135923) [ 7524.441618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7529.034882] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7537.088055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7543.489745] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7550.589681] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7556.869193] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7564.134878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7570.657529] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7578.349451] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7592.650689] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:39:57 (1787135997) [ 7598.575969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7598.581934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7598.594839] Lustre: Skipped 2 previous similar messages [ 7598.595904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7598.595911] Lustre: Skipped 3 previous similar messages [ 7598.611611] LustreError: Skipped 1 previous similar message [ 7602.996488] Lustre: server umount lustre-OST0000 complete [ 7612.767904] Lustre: server umount lustre-MDT0000 complete [ 7616.492242] LustreError: 174826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136023 with bad export cookie 4347961284441350327 [ 7616.507481] LustreError: 174826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7616.514155] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7616.802697] Lustre: server umount lustre-MDT0001 complete [ 7627.236335] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7629.877664] Key type lgssc unregistered [ 7630.232890] LNet: 178574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7630.240754] LNetError: 178574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7630.262877] LNet: Removed LNI 192.168.201.136@tcp [ 7631.142395] Key type .llcrypt unregistered [ 7631.144927] Key type ._llcrypt unregistered [ 7649.248254] Key type ._llcrypt registered [ 7649.250432] Key type .llcrypt registered [ 7649.334252] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7650.446325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7650.461645] alg: No test for adler32 (adler32-zlib) [ 7651.451284] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 7651.690928] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 7653.321073] Key type lgssc registered [ 7654.271517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7660.854233] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7660.883253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7676.384584] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7681.504377] LustreError: 179822:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.136@tcp: failed processing log, type 4: rc = -110 [ 7686.626333] LustreError: 179822:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7686.657572] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7686.673157] LustreError: MGC192.168.201.136@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7686.701450] LustreError: 179822:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7686.718520] LustreError: 179822:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7686.733909] LustreError: 179822:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7686.825406] Lustre: server umount lustre-OST0000 complete [ 7686.831797] LustreError: 179822:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7695.106765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7696.782112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.356737] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7706.087324] LustreError: 180320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7706.112468] LustreError: 180320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7710.439385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7710.833676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7715.490141] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7724.781191] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7731.451026] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7739.733682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7740.007716] LustreError: 182138:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7740.015545] Lustre: 182138:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.136@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7740.164299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7746.844982] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7754.399786] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7756.770116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7756.781805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.795353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7760.868955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7760.873584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7760.884421] Lustre: Skipped 1 previous similar message [ 7760.911505] Lustre: Skipped 2 previous similar messages [ 7763.090060] Lustre: server umount lustre-MDT0000 complete [ 7765.986358] LustreError: 181322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7766.015883] LustreError: 181322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7767.108291] LustreError: 182165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136174 with bad export cookie 13484138258833901560 [ 7767.115179] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7767.120442] LustreError: 182165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7767.445372] Lustre: server umount lustre-MDT0001 complete [ 7782.875490] Lustre: server umount lustre-OST0000 complete [ 7793.078866] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 7805.002261] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7843.781821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7844.070222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7844.152782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7844.260277] Lustre: lustre-MDT0000: new disk, initializing [ 7844.373130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7844.405784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7849.317781] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7861.565514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7861.727304] Lustre: 185664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7861.778491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7861.781853] Lustre: Skipped 1 previous similar message [ 7861.848917] Lustre: lustre-MDT0001: new disk, initializing [ 7862.003784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7862.070913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7862.084936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7867.401534] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7878.801098] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7884.363427] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7891.737746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7892.048722] Lustre: lustre-OST0000: new disk, initializing [ 7892.052770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7892.061040] Lustre: 187631:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7892.242524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.926237] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7900.225265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7900.232867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7900.382508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7909.056856] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7914.065531] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7914.326344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7918.892287] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7919.118295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7926.260775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7926.261994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.264803] LustreError: Skipped 1 previous similar message [ 7926.265887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7926.321273] Lustre: Skipped 1 previous similar message [ 7927.790796] Lustre: server umount lustre-OST0000 complete [ 7933.921454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7933.932416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.954847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7933.957487] Lustre: Skipped 1 previous similar message [ 7938.755115] Lustre: server umount lustre-MDT0000 complete [ 7941.608556] LustreError: 185674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7941.632119] LustreError: 185674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7944.019728] LustreError: 185654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136351 with bad export cookie 13484138258833902281 [ 7944.021560] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7944.035022] LustreError: 185654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7944.535124] Lustre: server umount lustre-MDT0001 complete [ 7955.463437] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:46:00 (1787136360) [ 7975.867700] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7978.704093] Key type lgssc unregistered [ 7979.004255] LNet: 190580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7979.021709] LNetError: 190580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7979.038461] LNet: Removed LNI 192.168.201.136@tcp [ 7980.072045] Key type .llcrypt unregistered [ 7980.074275] Key type ._llcrypt unregistered [ 7998.739152] Key type ._llcrypt registered [ 7998.740525] Key type .llcrypt registered [ 7998.813523] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8000.171470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8000.245807] alg: No test for adler32 (adler32-zlib) [ 8001.474984] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 8001.867665] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8003.697675] Key type lgssc registered [ 8004.716778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8012.272674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8012.296951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.731771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.435880] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8023.010900] LustreError: 191799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8023.027135] LustreError: 191799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8027.588358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8028.095161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8032.558528] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8039.841628] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8047.035884] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8056.065934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8056.312722] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8056.320215] LustreError: 193664:0:(llog.c:1696:llog_backup()) MGC192.168.201.136@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8056.339876] Lustre: 193664:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.136@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8056.573946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8064.240833] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8073.349232] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8079.331325] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8079.337294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8079.344410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8079.359101] Lustre: Skipped 2 previous similar messages [ 8079.382020] Lustre: Skipped 2 previous similar messages [ 8084.450376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8084.469184] Lustre: Skipped 2 previous similar messages [ 8089.571411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8089.583907] Lustre: Skipped 2 previous similar messages [ 8090.080138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8090.327373] Lustre: server umount lustre-MDT0000 complete [ 8094.692058] LustreError: 192535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8094.716366] LustreError: 192535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8094.923551] LustreError: 191779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136501 with bad export cookie 7212953125940541614 [ 8094.942420] LustreError: 191779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8094.945220] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8095.398119] Lustre: server umount lustre-MDT0001 complete [ 8109.893335] Lustre: server umount lustre-OST0000 complete [ 8120.763166] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 8130.734663] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8168.712396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8168.942960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8169.026892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8169.083329] Lustre: lustre-MDT0000: new disk, initializing [ 8169.136862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.150645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8174.196917] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8185.466651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.596793] Lustre: 197192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8185.632039] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8185.634071] Lustre: Skipped 1 previous similar message [ 8185.636671] LustreError: 197974:0:(llog.c:1696:llog_backup()) MGC192.168.201.136@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8185.642571] LustreError: 197974:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8185.646057] Lustre: 197974:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.136@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8185.661625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8185.665378] Lustre: Skipped 1 previous similar message [ 8185.732861] Lustre: lustre-MDT0001: new disk, initializing [ 8185.842925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8185.877382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8185.899657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8191.123757] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8202.202654] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8208.573448] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8216.121310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8216.453162] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8216.457030] Lustre: Skipped 1 previous similar message [ 8216.460655] LustreError: 199134:0:(llog.c:1696:llog_backup()) MGC192.168.201.136@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8216.473238] LustreError: 199134:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8216.482854] Lustre: 199134:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.136@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8216.498201] Lustre: 199134:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8216.518781] Lustre: lustre-OST0000: new disk, initializing [ 8216.526026] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8216.532206] Lustre: 199158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8216.654061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8222.314645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8222.328820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8222.550412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8224.472437] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8233.630932] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8237.684102] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8237.949148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8243.027349] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8243.278505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8248.296859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.302246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8248.310688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8248.319660] LustreError: Skipped 1 previous similar message [ 8251.552788] Lustre: server umount lustre-OST0000 complete [ 8257.507129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8257.526725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.553488] Lustre: Skipped 1 previous similar message [ 8257.575657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8257.589204] Lustre: Skipped 1 previous similar message [ 8268.773320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8268.798845] Lustre: Skipped 5 previous similar messages [ 8270.306054] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8270.527633] Lustre: server umount lustre-MDT0000 complete [ 8273.918735] LustreError: 199162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8273.938654] LustreError: 199162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8274.937967] LustreError: 200700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136681 with bad export cookie 7212953125940542342 [ 8274.947544] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8274.955117] LustreError: 200700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8275.338271] Lustre: server umount lustre-MDT0001 complete [ 8285.306975] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:51:30 (1787136690) [ 8303.282592] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8314.154342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8314.497440] LustreError: 202290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8314.539521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8319.278147] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8319.970848] LustreError: 202291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8327.985877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.232401] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8337.772160] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8344.897386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8352.066550] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8361.062370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8361.267485] Lustre: lustre-OST0001: new disk, initializing [ 8361.270972] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8361.279798] Lustre: 204598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8361.376932] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8361.388921] Lustre: Skipped 2 previous similar messages [ 8368.127864] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8368.269432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8368.281883] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8368.413052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8378.718714] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8389.365581] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8391.382306] Lustre: DEBUG MARKER: ADD OST3 [ 8396.415296] loop: module loaded [ 8396.513119] loop0: detected capacity change from 0 to 204800000 [ 8396.864415] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8396.875300] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8396.887486] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8403.230604] loop0: detected capacity change from 0 to 204800000 [ 8403.818297] Lustre: DEBUG MARKER: STOP OST3 [ 8408.819233] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8409.055137] Lustre: lustre-OST0003: new disk, initializing [ 8409.067445] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8409.078714] Lustre: 206606:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8410.820728] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8411.065721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8411.079647] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8411.122646] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8412.858520] LustreError: 206759:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8413.143944] Lustre: Failing over lustre-OST0003 [ 8413.224950] Lustre: server umount lustre-OST0003 complete [ 8415.680689] LustreError: 203779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8415.707544] LustreError: 203779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8416.230360] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8416.241249] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8416.256539] Lustre: Skipped 2 previous similar messages [ 8416.535668] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8425.380153] Lustre: DEBUG MARKER: START OST3 again [ 8433.608353] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8433.887296] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8433.890518] Lustre: Skipped 1 previous similar message [ 8433.908347] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8435.305414] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8435.977018] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8435.992045] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8441.068110] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8462.014297] Lustre: Failing over lustre-OST0003 [ 8462.249382] Lustre: server umount lustre-OST0003 complete [ 8464.904773] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.910024] LustreError: 204204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.936534] Lustre: Skipped 1 previous similar message [ 8464.976263] LustreError: 204204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8482.273513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8482.287088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.312944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8487.453628] Lustre: server umount lustre-MDT0000 complete [ 8491.167800] LustreError: 203425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787136898 with bad export cookie 7212953125940543217 [ 8491.174347] LustreError: 203425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8491.176969] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8497.602247] Lustre: server umount lustre-MDT0001 complete [ 8501.169690] Lustre: server umount lustre-OST0000 complete [ 8505.173130] Lustre: server umount lustre-OST0001 complete [ 8512.674503] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 8523.139483] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8571.987508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8572.169664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8572.193774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8572.245785] Lustre: lustre-MDT0000: new disk, initializing [ 8572.298549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8572.314300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8576.444717] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8587.509302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.694420] Lustre: 212059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8588.076845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8592.422324] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8600.992825] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8607.758069] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8615.871467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8616.157852] Lustre: lustre-OST0000: new disk, initializing [ 8616.160796] Lustre: Skipped 1 previous similar message [ 8616.165857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8616.175828] Lustre: Skipped 2 previous similar messages [ 8616.188993] Lustre: 214031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8617.772192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8617.797834] Lustre: Skipped 1 previous similar message [ 8617.810367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8618.028140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8625.220543] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8638.230412] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8644.558740] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8644.741171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8649.792696] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8650.043045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8653.794402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8653.799160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.805992] LustreError: Skipped 1 previous similar message [ 8653.817227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8653.840525] Lustre: Skipped 6 previous similar messages [ 8653.875659] Lustre: Skipped 9 previous similar messages [ 8658.253644] Lustre: server umount lustre-OST0000 complete [ 8676.320168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8676.565405] Lustre: server umount lustre-MDT0000 complete [ 8678.370430] LustreError: 212073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8678.398419] LustreError: 212073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8680.089060] LustreError: 212053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137087 with bad export cookie 7212953125940545520 [ 8680.093920] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8680.104487] LustreError: 212053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8680.562234] Lustre: server umount lustre-MDT0001 complete [ 8684.466850] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:58:09 (1787137089) [ 8703.786504] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 8715.234398] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8755.413216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8755.775911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8755.853166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8755.927896] Lustre: lustre-MDT0000: new disk, initializing [ 8756.029525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8761.118033] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8771.791432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8772.031942] Lustre: 218642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8772.331931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8778.112538] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8788.299549] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8794.542185] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8804.068940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8804.348448] Lustre: 220615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8805.966051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8806.185477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8812.327338] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8821.891744] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8827.006748] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8827.279305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8831.985726] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8832.182518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8837.091856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8837.094905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8837.095317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8837.095322] Lustre: Skipped 6 previous similar messages [ 8837.105965] Lustre: Skipped 3 previous similar messages [ 8848.352663] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8848.763554] Lustre: server umount lustre-OST0000 complete [ 8858.953193] Lustre: server umount lustre-MDT0000 complete [ 8862.041663] LustreError: 219438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137269 with bad export cookie 7212953125940546402 [ 8862.059250] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8862.369181] Lustre: server umount lustre-MDT0001 complete [ 8871.076701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8871.512745] LustreError: 222843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8871.652400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8871.662577] Lustre: Skipped 5 previous similar messages [ 8876.180961] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8884.115610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8889.756953] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8897.801982] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8904.459247] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8913.032921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8920.788681] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8927.952387] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8940.513831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8940.517309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8940.529620] Lustre: Skipped 3 previous similar messages [ 8943.960984] Lustre: server umount lustre-OST0000 complete [ 8954.170681] Lustre: server umount lustre-MDT0000 complete [ 8958.100943] LustreError: 222824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137365 with bad export cookie 7212953125940547284 [ 8958.116219] LustreError: 222824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8958.119666] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8958.545648] Lustre: server umount lustre-MDT0001 complete [ 9069.799896] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9072.883440] Key type lgssc unregistered [ 9073.253453] LNet: 226507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9073.284952] LNetError: 226507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9073.307369] LNet: Removed LNI 192.168.201.136@tcp [ 9074.120194] Key type .llcrypt unregistered [ 9074.121956] Key type ._llcrypt unregistered [ 9101.507490] Key type ._llcrypt registered [ 9101.510261] Key type .llcrypt registered [ 9101.651525] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9116.239913] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9117.347636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9117.494052] alg: No test for adler32 (adler32-zlib) [ 9118.624359] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 9118.951681] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9120.706043] Key type lgssc registered [ 9122.072900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9160.986624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9161.039095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9162.333678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9162.397516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9162.523086] Lustre: lustre-MDT0000: new disk, initializing [ 9162.677603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9162.746905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9167.677592] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9177.919602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.018435] Lustre: 229858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9178.060957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9178.064247] Lustre: Skipped 1 previous similar message [ 9178.125337] Lustre: lustre-MDT0001: new disk, initializing [ 9178.191442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9178.233490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9178.257254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9182.299449] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9191.340702] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9196.749891] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9203.654611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9203.960455] Lustre: lustre-OST0000: new disk, initializing [ 9203.967968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9203.981970] Lustre: 231827:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9204.122818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9210.618735] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9211.448076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9211.463645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9211.571079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9220.185742] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9224.974906] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9225.135562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9229.290157] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9229.480818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9232.361776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9232.364451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9232.367158] LustreError: Skipped 1 previous similar message [ 9232.370686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9232.416996] Lustre: Skipped 1 previous similar message [ 9237.374746] Lustre: server umount lustre-OST0000 complete [ 9242.140090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9242.148975] Lustre: Skipped 1 previous similar message [ 9242.156285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9242.165438] Lustre: Skipped 2 previous similar messages [ 9247.204155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9247.210440] Lustre: Skipped 1 previous similar message [ 9248.444848] Lustre: server umount lustre-MDT0000 complete [ 9252.709675] LustreError: 229850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137659 with bad export cookie 4213841470470658847 [ 9252.725549] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9252.729788] LustreError: 229850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9253.057574] Lustre: server umount lustre-MDT0001 complete [ 9274.188986] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9283.577630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9283.913560] LustreError: 234764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9283.942673] LustreError: 234764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9283.987485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.082446] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9289.186988] LustreError: 234765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9293.288227] LustreError: 234764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9298.404965] LustreError: 234765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9299.068830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.462155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9304.618518] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9307.969750] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9314.905109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9315.203297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9322.128525] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9332.339865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9332.646073] Lustre: lustre-OST0001: new disk, initializing [ 9332.650806] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9332.655942] Lustre: 237074:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9332.780379] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9338.468591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9338.479270] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9338.607555] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9339.514330] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9350.755509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9356.872311] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9369.500487] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:09:34 (1787137774) [ 9374.709055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9374.720532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9374.745676] Lustre: Skipped 2 previous similar messages [ 9374.758624] Lustre: Skipped 1 previous similar message [ 9378.318251] Lustre: server umount lustre-MDT0000 complete [ 9379.823847] LustreError: 234765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9379.854830] LustreError: 234765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9383.509176] LustreError: 234745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137790 with bad export cookie 4213841470470659729 [ 9383.527623] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9383.529552] LustreError: 234745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9384.134913] Lustre: server umount lustre-MDT0001 complete [ 9399.373481] Lustre: server umount lustre-OST0000 complete [ 9400.096191] Lustre: 228083:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787137791/real 1787137791] req@ffff8ee3092c5880 x1873949517526656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787137807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9400.108101] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9400.123891] Lustre: Skipped 1 previous similar message [ 9403.597410] Lustre: server umount lustre-OST0001 complete [ 9409.912730] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9421.738808] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9462.143669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.326838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9462.366423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9462.465094] Lustre: lustre-MDT0000: new disk, initializing [ 9462.543461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9462.554511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9468.181754] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9479.109819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9479.268908] Lustre: 241016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9479.367403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9479.374792] Lustre: Skipped 1 previous similar message [ 9479.505719] Lustre: lustre-MDT0001: new disk, initializing [ 9479.614558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9479.628175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9485.602744] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9496.587292] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9502.832365] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9510.515161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9510.792777] Lustre: lustre-OST0000: new disk, initializing [ 9510.798211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9510.803922] Lustre: 242986:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9512.593730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9512.601125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9512.714671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9517.126869] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9527.091167] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9532.772888] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9533.076664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9538.628706] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9538.903652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9543.649411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9543.656147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9543.669991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9543.711700] Lustre: Skipped 1 previous similar message [ 9547.404709] Lustre: server umount lustre-OST0000 complete [ 9553.892969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9553.898655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9553.917433] Lustre: Skipped 1 previous similar message [ 9553.936720] Lustre: Skipped 2 previous similar messages [ 9558.079953] Lustre: server umount lustre-MDT0000 complete [ 9559.009373] LustreError: 241027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9559.032321] LustreError: 241027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9563.396962] LustreError: 241007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787137970 with bad export cookie 4213841470470661864 [ 9563.419073] LustreError: 241007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9563.425595] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9563.925047] Lustre: server umount lustre-MDT0001 complete [ 9575.264241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9576.034868] LustreError: 245202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9576.055638] LustreError: 245202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9576.135900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9576.145092] Lustre: Skipped 2 previous similar messages [ 9581.720641] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9590.571225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9596.196248] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9605.512328] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9613.274418] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9621.100059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9627.853568] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9636.244217] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9784.302583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9784.309333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9784.310057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9784.315629] Lustre: Skipped 1 previous similar message [ 9789.624710] Lustre: server umount lustre-OST0000 complete [ 9799.899627] Lustre: server umount lustre-MDT0000 complete [ 9803.972357] LustreError: 245183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138211 with bad export cookie 4213841470470662746 [ 9803.983452] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9804.252510] Lustre: server umount lustre-MDT0001 complete [ 9906.384500] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9910.803474] Key type lgssc unregistered [ 9911.150577] LNet: 248811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9911.161757] LNetError: 248811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9911.176868] LNet: Removed LNI 192.168.201.136@tcp [ 9912.281180] Key type .llcrypt unregistered [ 9912.282736] Key type ._llcrypt unregistered [ 9940.983736] Key type ._llcrypt registered [ 9940.988595] Key type .llcrypt registered [ 9941.080568] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9959.625066] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9960.888222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9960.977209] alg: No test for adler32 (adler32-zlib) [ 9961.989978] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 9962.256176] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9963.992380] Key type lgssc registered [ 9965.188871] Lustre: Echo OBD driver; http://www.lustre.org/ [10009.391178] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10019.565647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10019.577384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10020.880389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10020.917328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10021.014861] Lustre: lustre-MDT0000: new disk, initializing [10021.095643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10021.115742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10025.579442] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10036.660408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10036.814780] Lustre: 252887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10036.861191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10036.864366] Lustre: Skipped 1 previous similar message [10036.951355] Lustre: lustre-MDT0001: new disk, initializing [10037.043542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10037.074518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10037.083088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10041.482752] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10046.287517] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10052.913530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10053.266458] Lustre: lustre-OST0000: new disk, initializing [10053.270149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10053.279797] Lustre: 254515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10053.397722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10055.388267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10055.394565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10055.541721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10059.375872] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10069.923505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10070.126190] Lustre: lustre-OST0001: new disk, initializing [10070.128961] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10070.134857] Lustre: 255388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10070.258188] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10076.392484] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10078.324602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10078.339601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10078.467145] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10089.022264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10100.426339] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10114.018074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10114.029638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10114.048216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10114.532391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10114.550156] Lustre: Skipped 1 previous similar message [10114.557431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10114.568052] Lustre: Skipped 2 previous similar messages [10119.651273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10119.655971] Lustre: Skipped 1 previous similar message [10124.788064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10124.798413] Lustre: Skipped 5 previous similar messages [10126.304133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10126.606179] Lustre: server umount lustre-MDT0000 complete [10130.329708] LustreError: 255386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138537 with bad export cookie 6277099397830277620 [10130.330973] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10130.344857] LustreError: 255386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10130.713433] Lustre: server umount lustre-MDT0001 complete [10145.367566] Lustre: server umount lustre-OST0000 complete [10159.693832] Lustre: server umount lustre-OST0001 complete [10176.814325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10187.191043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10195.607127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10206.167858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10226.040580] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10236.002331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.017599] Lustre: lustre-MDT0000: reset Object Index mappings [10236.562348] LustreError: 260403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10236.572908] LustreError: 260403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10236.645085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10240.488885] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10242.020896] LustreError: 260404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10246.126482] LustreError: 260403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10248.737726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10248.753916] Lustre: lustre-MDT0001: reset Object Index mappings [10249.220179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10253.868612] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10257.035503] Lustre: 261544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10264.763706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10265.344393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10272.644678] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10274.535877] LustreError: 261897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10279.925164] LustreError: 261898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10282.018471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10282.510803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10288.150253] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10290.404445] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10298.486381] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10301.980260] Lustre: 263418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10308.583539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10308.596427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10308.612086] Lustre: Skipped 3 previous similar messages [10308.632182] Lustre: Skipped 3 previous similar messages [10311.070787] Lustre: server umount lustre-MDT0000 complete [10313.709670] LustreError: 260398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10313.725171] LustreError: 260398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10314.980760] LustreError: 263436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138722 with bad export cookie 6277099397830284410 [10315.003983] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10315.623076] Lustre: server umount lustre-MDT0001 complete [10330.566208] Lustre: server umount lustre-OST0000 complete [10334.180620] Lustre: 250393:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787138725/real 1787138725] req@ffff8ee30aeeb480 x1873950402544640/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787138741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10334.213731] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10334.230406] Lustre: Skipped 1 previous similar message [10335.355979] Lustre: server umount lustre-OST0001 complete [10358.330856] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:26:03 (1787138763) [10376.576418] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [10388.539696] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10425.798598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10426.160868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10426.210446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10426.378247] Lustre: lustre-MDT0000: new disk, initializing [10426.490734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10426.497837] Lustre: Skipped 1 previous similar message [10426.515241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10431.032854] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10442.269034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10442.454387] Lustre: 267671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10442.507093] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10442.517522] Lustre: Skipped 1 previous similar message [10442.637773] Lustre: lustre-MDT0001: new disk, initializing [10442.862490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10442.873431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10447.733588] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10459.195145] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10464.205594] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10470.845673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10471.254913] Lustre: lustre-OST0000: new disk, initializing [10471.260917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10471.269846] Lustre: 269642:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10473.220843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10473.234941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10473.387148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10477.131915] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10487.120722] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10492.380856] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10492.581137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10496.895623] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10497.186887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10504.165466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10504.167343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10504.177229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10504.183161] LustreError: Skipped 1 previous similar message [10504.226226] Lustre: Skipped 1 previous similar message [10505.575757] Lustre: server umount lustre-OST0000 complete [10514.402436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10516.538686] Lustre: server umount lustre-MDT0000 complete [10519.525799] LustreError: 269548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10519.552541] LustreError: 269548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10520.568209] LustreError: 267663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787138927 with bad export cookie 6277099397830286503 [10520.576108] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10520.580516] LustreError: 267663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10521.147232] Lustre: server umount lustre-MDT0001 complete [10547.243809] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10559.765880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.427961] LustreError: 272594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10560.488478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10560.496481] Lustre: Skipped 2 previous similar messages [10566.103726] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10575.754094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10580.980488] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10584.007739] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10593.428375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10601.136167] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10612.476542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10612.685819] Lustre: lustre-OST0001: new disk, initializing [10612.689875] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10612.696047] Lustre: 274907:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10614.479526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10614.510379] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10614.764831] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10621.210771] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10632.994397] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10638.825323] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10645.645876] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10645.822662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10655.740984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10655.757990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10655.769060] Lustre: Skipped 3 previous similar messages [10655.790819] Lustre: Skipped 6 previous similar messages [10657.707157] Lustre: server umount lustre-MDT0000 complete [10660.834354] LustreError: 274005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10660.882245] LustreError: 274005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10662.900533] LustreError: 274103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787139069 with bad export cookie 6277099397830287385 [10662.911508] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10662.917064] LustreError: 274103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10663.485498] Lustre: server umount lustre-MDT0001 complete [10678.155964] Lustre: server umount lustre-OST0000 complete [10682.337579] Lustre: 250394:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787139073/real 1787139073] req@ffff8ee307d04a80 x1873950402662912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787139089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10683.021471] Lustre: server umount lustre-OST0001 complete [10705.156521] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10719.495545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10720.066787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10720.083639] Lustre: Skipped 3 previous similar messages [10724.689372] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10734.343136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10740.427509] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10744.623951] Lustre: 279108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10750.646121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10759.262711] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10765.354131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10770.873664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10776.590264] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10778.225773] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10787.019918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10792.004211] Lustre: 280985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10799.140935] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10799.165416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10813.721150] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:33:38 (1787139218) [10831.841339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10831.854766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10831.878190] Lustre: Skipped 2 previous similar messages [10831.890041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10837.988137] LustreError: 277961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10838.001553] LustreError: 277961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [10838.024430] Lustre: server umount lustre-MDT0000 complete [10841.647521] LustreError: 277947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787139248 with bad export cookie 6277099397830294161 [10841.669142] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10841.671301] LustreError: 277947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10848.339467] Lustre: server umount lustre-MDT0001 complete [10859.442945] Lustre: server umount lustre-OST0000 complete [10865.401410] Lustre: server umount lustre-OST0001 complete [10873.904191] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [10886.882942] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10940.867990] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10952.136492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10952.412975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10952.435320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10952.506336] Lustre: lustre-MDT0000: new disk, initializing [10952.647159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10957.426800] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10968.028595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10968.153148] Lustre: 285232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10968.410486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10972.994842] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10984.828586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10985.134770] Lustre: 286865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10985.238146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10985.248128] Lustre: Skipped 5 previous similar messages [10987.050220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10991.241016] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11002.918988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11003.211191] Lustre: 287738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11004.937942] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11004.947357] Lustre: Skipped 1 previous similar message [11005.101816] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11011.565639] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11022.759223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11027.006420] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11027.010735] Lustre: Skipped 1 previous similar message [11045.687704] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:37:30 (1787139450) [11050.482118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11050.496745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11050.508900] Lustre: Skipped 5 previous similar messages [11050.514773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11050.524601] Lustre: Skipped 8 previous similar messages [11050.799262] Lustre: server umount lustre-MDT0000 complete [11055.534914] LustreError: 285224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787139462 with bad export cookie 6277099397830301602 [11055.542915] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11055.543755] LustreError: 285224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11055.708201] Lustre: server umount lustre-MDT0001 complete [11071.200194] Lustre: server umount lustre-OST0000 complete [11072.934858] Lustre: 250392:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787139463/real 1787139463] req@ffff8ee30e2a9500 x1873950402824448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787139479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11076.224069] Lustre: server umount lustre-OST0001 complete [11085.257745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11090.849276] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11095.016141] LustreError: 289983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11095.048411] LustreError: 289983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11100.186347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11105.792587] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11113.934253] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11119.806592] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11127.818678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11137.074441] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11145.738924] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11163.256641] loop1: detected capacity change from 0 to 204800000 [11163.443286] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11163.457272] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11163.465945] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11168.211197] loop1: detected capacity change from 0 to 204800000 [11174.587427] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11174.778352] LustreError: 289972:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs7153: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11174.797477] LustreError: 289972:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs7153 [11174.804816] LustreError: 289972:0:(mgs_handler.c:517:mgs_target_reg()) fs7153-MDT0001: Error checking target index: rc = -13 [11174.822935] LustreError: fs7153-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11174.841222] LustreError: 293699:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11174.853436] LustreError: 293699:0:(tgt_mount.c:2129:server_put_super()) no obd fs7153-MDT0001 [11174.864332] LustreError: 293699:0:(tgt_mount.c:116:server_deregister_mount()) fs7153-MDT0001 not registered [11174.901578] Lustre: server umount fs7153-MDT0001 complete [11174.904681] LustreError: 293699:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11183.853592] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11183.985624] Lustre: Setting parameter fs7153-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs7153-MDT0001 [11184.027642] Lustre: srv-fs7153-MDT0001: No data found on store. Initialize space. [11184.031481] Lustre: Skipped 4 previous similar messages [11184.098304] Lustre: fs7153-MDT0001: new disk, initializing [11184.102136] Lustre: Skipped 3 previous similar messages [11184.192749] Lustre: srv-fs7153-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11184.197268] Lustre: 294167:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7153-MDT0001-mdtlov: can't create new llog: rc = -115 [11184.207912] LustreError: 294167:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7153-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11188.182461] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11195.194820] Lustre: server umount fs7153-MDT0001 complete [11210.232307] loop1: detected capacity change from 0 to 204800000 [11212.281762] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11217.163445] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11217.774655] Lustre: 295734:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7153-MDT0001-mdtlov: can't create new llog: rc = -115 [11217.788910] LustreError: 295734:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7153-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11224.127447] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11227.649986] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11230.463915] Lustre: server umount fs7153-MDT0001 complete [11242.983786] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11242.988948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11242.996824] Lustre: Skipped 6 previous similar messages [11243.013379] Lustre: Skipped 2 previous similar messages [11247.958773] Lustre: server umount lustre-MDT0000 complete [11258.233691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11258.375362] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11263.997062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11264.169768] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11272.444365] loop1: detected capacity change from 0 to 204800000 [11278.248039] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11278.672501] Lustre: 298182:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7153-MDT0001-mdtlov: can't create new llog: rc = -115 [11278.686692] LustreError: 298182:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7153-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11283.430524] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11288.382225] Lustre: server umount fs7153-MDT0001 complete [11309.541818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11315.729604] Lustre: server umount lustre-OST0000 complete [11326.235729] Lustre: server umount lustre-MDT0000 complete [11331.532127] LustreError: 289963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787139738 with bad export cookie 6277099397830304283 [11331.541382] LustreError: 289963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11331.553815] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11331.861252] Lustre: server umount lustre-MDT0001 complete [11344.918780] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11348.451429] Key type lgssc unregistered [11348.872788] LNet: 300528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11348.885871] LNetError: 300528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11349.941475] LNet: Removed LNI 192.168.201.136@tcp [11351.023222] Key type .llcrypt unregistered [11351.025122] Key type ._llcrypt unregistered [11369.864649] Key type ._llcrypt registered [11369.867942] Key type .llcrypt registered [11369.979279] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11371.335427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11371.365855] alg: No test for adler32 (adler32-zlib) [11372.442398] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [11372.640839] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11374.337075] Key type lgssc registered [11375.589054] Lustre: Echo OBD driver; http://www.lustre.org/ [11383.862136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11383.877147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11385.310213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11390.041214] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11395.559519] LustreError: 301801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11395.592160] LustreError: 301801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11398.118507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11398.430031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11402.532400] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11411.646982] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11417.786209] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11424.844348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11425.280483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11430.386795] LustreError: 303631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11432.731648] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11435.495581] LustreError: 303631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.619472] LustreError: 303897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.650562] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11448.222117] LustreError: 303629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11448.256578] LustreError: 303629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11458.456428] LustreError: 303630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11458.475427] LustreError: 303630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11460.691782] loop1: detected capacity change from 0 to 204800000 [11460.901882] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11460.914315] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11460.928320] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11466.067666] loop1: detected capacity change from 0 to 204800000 [11472.036896] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11472.211757] Lustre: fs20784-OST0000: new disk, initializing [11472.218825] Lustre: srv-fs20784-OST0000: No data found on store. Initialize space. [11472.225368] Lustre: 305535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11472.322050] Lustre: fs20784-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11478.971996] LustreError: 303630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11478.996890] LustreError: 303630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [11479.586741] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11487.590793] Lustre: server umount fs20784-OST0000 complete [11498.761483] loop1: detected capacity change from 0 to 204800000 [11504.813505] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11505.224591] Lustre: fs20784-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11510.243805] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11511.731430] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11514.798585] LustreError: 303630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11514.826125] LustreError: 303630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [11521.088664] loop2: detected capacity change from 0 to 204800000 [11521.347421] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11521.362093] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11521.400015] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11525.634074] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11527.295688] loop2: detected capacity change from 0 to 204800000 [11534.497382] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11534.581907] LustreError: fs20784-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11534.591959] LustreError: fs20784-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11534.607609] LustreError: 308577:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11534.615621] LustreError: 308577:0:(tgt_mount.c:2129:server_put_super()) no obd fs20784-OST0001 [11534.620404] LustreError: 308577:0:(tgt_mount.c:116:server_deregister_mount()) fs20784-OST0001 not registered [11534.670586] Lustre: server umount fs20784-OST0001 complete [11534.673931] LustreError: 308577:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11548.207612] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11548.332882] Lustre: fs20784-OST0001: new disk, initializing [11548.336736] Lustre: srv-fs20784-OST0001: No data found on store. Initialize space. [11548.346623] Lustre: 309129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11548.425283] Lustre: fs20784-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11555.254598] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11563.567075] Lustre: server umount fs20784-OST0000 complete [11573.307877] Lustre: server umount fs20784-OST0001 complete [11581.354497] LustreError: 303631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11581.382255] LustreError: 303631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11584.419152] loop1: detected capacity change from 0 to 204800000 [11589.810614] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11590.149678] Lustre: fs20784-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11595.937166] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11602.601968] loop2: detected capacity change from 0 to 204800000 [11607.939346] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11614.280723] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11622.757957] Lustre: server umount fs20784-OST0000 complete [11632.775826] Lustre: server umount fs20784-OST0001 complete [11645.413946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11645.429046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11645.445918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11648.482830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11648.484263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11651.461732] Lustre: server umount lustre-OST0000 complete [11658.734265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11658.749444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11658.756921] Lustre: Skipped 2 previous similar messages [11661.687781] Lustre: server umount lustre-MDT0000 complete [11665.878942] LustreError: 306282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140072 with bad export cookie 9896301525713572222 [11665.887356] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11665.898172] LustreError: 306282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11666.211160] Lustre: server umount lustre-MDT0001 complete [11676.818329] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11679.552710] Key type lgssc unregistered [11679.815793] LNet: 315053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11679.826340] LNetError: 315053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11679.854031] LNet: Removed LNI 192.168.201.136@tcp [11680.781250] Key type .llcrypt unregistered [11680.786081] Key type ._llcrypt unregistered [11711.658694] Key type ._llcrypt registered [11711.662560] Key type .llcrypt registered [11711.837398] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [11728.493749] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11730.209393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11730.291331] alg: No test for adler32 (adler32-zlib) [11731.400090] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [11731.691601] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11733.480198] Key type lgssc registered [11734.625894] Lustre: Echo OBD driver; http://www.lustre.org/ [11775.173785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11775.196129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11776.522370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11776.556988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11776.651468] Lustre: lustre-MDT0000: new disk, initializing [11776.785632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11776.820943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11782.845975] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11794.968653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11795.127551] Lustre: 318913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11795.201620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11795.205736] Lustre: Skipped 1 previous similar message [11795.280561] Lustre: lustre-MDT0001: new disk, initializing [11795.403654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11795.480726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11795.502420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11800.456259] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11812.613265] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11819.926645] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11828.878367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11829.400725] Lustre: lustre-OST0000: new disk, initializing [11829.405503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11829.411872] Lustre: 320880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11829.516598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11838.046092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11838.059651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11838.067224] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11838.252712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11850.656767] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11853.699614] Lustre: Mounted lustre-client [11854.819310] Lustre: Unmounted lustre-client [11869.159802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11869.164423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11869.178275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11869.190747] Lustre: Skipped 1 previous similar message [11869.223694] LustreError: Skipped 1 previous similar message [11872.447703] Lustre: server umount lustre-OST0000 complete [11877.349915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11877.356053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11877.378222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11877.383362] Lustre: Skipped 1 previous similar message [11879.395692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11879.403659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11882.811802] Lustre: server umount lustre-MDT0000 complete [11887.297809] LustreError: 318906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140294 with bad export cookie 5407377207074974831 [11887.306100] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11887.316524] LustreError: 318906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11887.741649] Lustre: server umount lustre-MDT0001 complete [11901.248570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11904.714666] Key type lgssc unregistered [11914.016266] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp [11914.337784] LNet: 323045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11914.351482] LNetError: 323045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11986.093802] LNet: Removed LNI 192.168.201.136@tcp [11987.472763] Key type .llcrypt unregistered [11987.474915] Key type ._llcrypt unregistered [12005.975452] Key type ._llcrypt registered [12005.977405] Key type .llcrypt registered [12006.118062] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [12023.032455] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12024.670396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12024.886117] alg: No test for adler32 (adler32-zlib) [12025.910988] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12026.089605] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12027.784149] Key type lgssc registered [12028.875579] Lustre: Echo OBD driver; http://www.lustre.org/ [12066.785906] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:54:32 (1787140472) [12076.382581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12076.418667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12077.785348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12077.878747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12078.134301] Lustre: lustre-MDT0000: new disk, initializing [12078.279521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12078.328149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12083.305076] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12094.758776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12094.881978] Lustre: 326271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12094.948667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12094.955475] Lustre: Skipped 1 previous similar message [12095.040797] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12095.057475] Lustre: lustre-MDT0001: new disk, initializing [12095.131209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12095.186373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12095.203354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12100.099411] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12110.928680] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12119.204992] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12127.418076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12127.818368] Lustre: lustre-OST0000: new disk, initializing [12127.822898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12127.832713] Lustre: 328242:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12127.946492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12133.417063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12133.429582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12133.533104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12135.070505] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12146.244317] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12164.577943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12164.578039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12164.593167] LustreError: Skipped 1 previous similar message [12164.601180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12164.604878] Lustre: Skipped 1 previous similar message [12164.622051] Lustre: Skipped 1 previous similar message [12167.336460] Lustre: server umount lustre-OST0000 complete [12174.821758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12174.825229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12174.845722] Lustre: Skipped 1 previous similar message [12174.882448] Lustre: Skipped 1 previous similar message [12179.691565] Lustre: server umount lustre-MDT0000 complete [12185.263616] LustreError: 326263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140592 with bad export cookie 8718088850159036832 [12185.275152] LustreError: 326263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12185.277175] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12185.714266] Lustre: server umount lustre-MDT0001 complete [12200.484650] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12203.485508] Key type lgssc unregistered [12203.790664] LNet: 330152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12203.795682] LNetError: 330152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12203.816500] LNet: Removed LNI 192.168.201.136@tcp [12204.910317] Key type .llcrypt unregistered [12204.915986] Key type ._llcrypt unregistered [12229.639127] Key type ._llcrypt registered [12229.641136] Key type .llcrypt registered [12229.856830] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12231.411425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12231.662189] alg: No test for adler32 (adler32-zlib) [12233.339560] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12233.718508] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12235.497310] Key type lgssc registered [12237.192536] Lustre: Echo OBD driver; http://www.lustre.org/ [12248.498154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12248.511946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12250.077763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12256.920763] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12260.332282] LustreError: 331428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.354475] LustreError: 331428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12265.452019] LustreError: 331429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.752314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.367702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12275.262805] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12285.552940] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12293.346386] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12302.476549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12302.804468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12311.476085] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12321.595984] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12334.355995] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12336.265023] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12340.709246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12340.712862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12340.724360] Lustre: Skipped 1 previous similar message [12340.739802] Lustre: Skipped 2 previous similar messages [12345.285629] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [12346.966182] Lustre: server umount lustre-MDT0000 complete [12350.362304] LustreError: 331425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12351.391898] LustreError: 331408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140758 with bad export cookie 16133464830714420675 [12351.404740] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12351.410717] LustreError: 331408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12355.486500] LustreError: 331425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12355.502913] LustreError: 331425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12355.521592] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [12355.525666] Lustre: Skipped 3 previous similar messages [12356.068590] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12356.096475] Lustre: Skipped 1 previous similar message [12357.880526] Lustre: server umount lustre-MDT0001 complete [12362.457469] Lustre: server umount lustre-OST0000 complete [12373.269935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12378.796742] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12382.169834] Lustre: 335613:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.136@tcp [12384.896981] Lustre: server umount MGS complete [12394.383588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12394.996152] LustreError: 336185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12395.027915] LustreError: 336185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12395.110486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12400.636583] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12405.225244] LustreError: 336186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12405.237096] LustreError: 336186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12410.308608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12410.728914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12415.809956] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12425.220962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12425.562434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12433.151884] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12442.130469] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12444.313440] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12451.108122] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12453.188834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12461.541125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12461.541241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12461.548959] LustreError: Skipped 1 previous similar message [12461.558080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12461.575718] Lustre: Skipped 1 previous similar message [12461.588123] Lustre: Skipped 1 previous similar message [12465.936462] Lustre: server umount lustre-OST0000 complete [12471.778285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12471.790421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12471.808480] Lustre: Skipped 1 previous similar message [12471.840585] Lustre: Skipped 2 previous similar messages [12476.671651] Lustre: server umount lustre-MDT0000 complete [12476.904551] LustreError: 336186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12476.944236] LustreError: 336186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12481.326058] LustreError: 336167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787140888 with bad export cookie 16133464830714421508 [12481.339691] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12481.344090] LustreError: 336167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12482.289807] Lustre: server umount lustre-MDT0001 complete [12498.041078] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12501.652497] Key type lgssc unregistered [12502.163068] LNet: 339631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12502.188551] LNetError: 339631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12502.204807] LNet: Removed LNI 192.168.201.136@tcp [12503.472565] Key type .llcrypt unregistered [12503.474507] Key type ._llcrypt unregistered [12526.671641] Key type ._llcrypt registered [12526.678312] Key type .llcrypt registered [12526.828903] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [12544.576886] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12545.826628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12546.032056] alg: No test for adler32 (adler32-zlib) [12547.123628] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12547.347328] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12549.002754] Key type lgssc registered [12550.159814] Lustre: Echo OBD driver; http://www.lustre.org/ [12589.930517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12589.999654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12591.272567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12591.305621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12591.372544] Lustre: lustre-MDT0000: new disk, initializing [12591.465423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12591.476799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12596.077362] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12607.594659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12607.707944] Lustre: 342802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12607.758747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12607.763298] Lustre: Skipped 1 previous similar message [12607.867307] Lustre: lustre-MDT0001: new disk, initializing [12608.037289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12608.109589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12608.133969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12612.902297] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12621.125627] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12627.304797] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12635.191574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12635.633575] Lustre: lustre-OST0000: new disk, initializing [12635.638735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12635.646571] Lustre: 344769:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12635.742428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12641.726472] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12641.909300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12641.927951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12642.133901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12650.614497] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12655.160334] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12655.366512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12660.065675] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12660.385091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12662.762969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12662.763271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12662.770216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12662.785406] Lustre: Skipped 1 previous similar message [12662.791049] LustreError: Skipped 1 previous similar message [12667.876496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12667.894137] Lustre: Skipped 1 previous similar message [12668.682979] Lustre: server umount lustre-OST0000 complete [12674.529370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12674.536112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12674.574285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12678.115146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12678.117931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12683.236662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12683.243344] Lustre: Skipped 2 previous similar messages [12688.353293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12688.629852] Lustre: server umount lustre-MDT0000 complete [12692.887466] LustreError: 342792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141099 with bad export cookie 15942808909274204642 [12692.893283] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12692.896641] LustreError: 342792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12693.209881] Lustre: server umount lustre-MDT0001 complete [12701.623314] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 08:05:07 (1787141107) [12716.065657] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [12726.093326] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12761.906847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12762.094563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12762.113477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12762.192837] Lustre: lustre-MDT0000: new disk, initializing [12762.345762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12762.360322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12767.286522] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12778.314808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12778.502437] Lustre: 349526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12778.565540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12778.569904] Lustre: Skipped 1 previous similar message [12778.587189] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12778.592211] Lustre: Skipped 2 previous similar messages [12778.626902] Lustre: lustre-MDT0001: new disk, initializing [12778.710333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12778.742882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12778.752962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12784.169057] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12794.338824] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12799.477222] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12808.614623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12808.924770] Lustre: lustre-OST0000: new disk, initializing [12808.937241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12808.942835] Lustre: 351497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12809.058291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12815.179372] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12816.462482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12816.472608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12816.589128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12824.743212] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12868.067254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12868.077020] LustreError: Skipped 1 previous similar message [12868.077045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12868.102319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12868.121294] Lustre: Skipped 1 previous similar message [12868.143060] Lustre: Skipped 1 previous similar message [12878.304318] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12878.503659] Lustre: server umount lustre-OST0000 complete [12883.439711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12883.463458] Lustre: Skipped 1 previous similar message [12889.720759] Lustre: server umount lustre-MDT0000 complete [12893.113501] LustreError: 349519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787141300 with bad export cookie 15942808909274205531 [12893.119414] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12893.124491] LustreError: 349519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12893.378447] Lustre: server umount lustre-MDT0001 complete [12903.426246] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12906.017903] Key type lgssc unregistered [12906.341781] LNet: 353671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12906.349825] LNetError: 353671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12906.380772] LNet: Removed LNI 192.168.201.136@tcp [12907.577310] Key type .llcrypt unregistered [12907.579350] Key type ._llcrypt unregistered [13012.232428] Key type ._llcrypt registered [13012.234818] Key type .llcrypt registered [13012.333794] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13012.531678] Key type .llcrypt unregistered [13012.535616] Key type ._llcrypt unregistered [13100.529548] Key type ._llcrypt registered [13100.532035] Key type .llcrypt registered [13100.693940] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13101.029137] Key type .llcrypt unregistered [13101.037069] Key type ._llcrypt unregistered [13135.344153] Key type ._llcrypt registered [13135.345457] Key type .llcrypt registered [13135.445167] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13135.812548] Key type .llcrypt unregistered [13135.818084] Key type ._llcrypt unregistered [13169.777401] Key type ._llcrypt registered [13169.784485] Key type .llcrypt registered [13169.929318] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13170.196346] Key type .llcrypt unregistered [13170.197785] Key type ._llcrypt unregistered [13248.122617] Key type ._llcrypt registered [13248.131543] Key type .llcrypt registered [13248.249850] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13248.569539] Key type .llcrypt unregistered [13248.577864] Key type ._llcrypt unregistered [13278.952573] Key type ._llcrypt registered [13278.954017] Key type .llcrypt registered [13279.061708] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13279.275674] Key type .llcrypt unregistered [13279.278532] Key type ._llcrypt unregistered [13313.080720] Key type ._llcrypt registered [13313.083558] Key type .llcrypt registered [13313.234276] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13313.554550] Key type .llcrypt unregistered [13313.556460] Key type ._llcrypt unregistered [13348.347258] Key type ._llcrypt registered [13348.352761] Key type .llcrypt registered [13348.518867] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13349.024052] Key type .llcrypt unregistered [13349.029331] Key type ._llcrypt unregistered [13384.379537] Key type ._llcrypt registered [13384.381954] Key type .llcrypt registered [13384.491429] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13384.789710] Key type .llcrypt unregistered [13384.801963] Key type ._llcrypt unregistered [13419.892592] Key type ._llcrypt registered [13419.896431] Key type .llcrypt registered [13420.111292] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13420.536567] Key type .llcrypt unregistered [13420.537892] Key type ._llcrypt unregistered [13494.221859] Key type ._llcrypt registered [13494.225424] Key type .llcrypt registered [13494.332763] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13494.706709] Key type .llcrypt unregistered [13494.713868] Key type ._llcrypt unregistered [13535.145158] Key type ._llcrypt registered [13535.148539] Key type .llcrypt registered [13535.288487] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13535.616318] Key type .llcrypt unregistered [13535.620495] Key type ._llcrypt unregistered [13569.898073] Key type ._llcrypt registered [13569.900743] Key type .llcrypt registered [13570.003277] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13570.287440] Key type .llcrypt unregistered [13570.289705] Key type ._llcrypt unregistered [13633.329819] Key type ._llcrypt registered [13633.331410] Key type .llcrypt registered [13633.479333] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [13653.771432] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [13655.543383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13655.648724] alg: No test for adler32 (adler32-zlib) [13656.820421] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [13657.058292] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [13658.760668] Key type lgssc registered [13659.963898] Lustre: Echo OBD driver; http://www.lustre.org/ [13700.930760] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13700.973725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13702.343174] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13702.391560] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13702.516845] Lustre: tst250fs-MDT0000: new disk, initializing [13702.614750] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13702.635634] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13707.680171] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13720.476441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13720.591359] Lustre: 374017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13720.649266] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13720.656759] Lustre: Skipped 1 previous similar message [13720.742394] Lustre: tst250fs-MDT0001: new disk, initializing [13720.853783] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13720.907596] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13720.924568] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13725.686786] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13731.336495] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13740.765834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13741.281459] Lustre: tst250fs-OST0000: new disk, initializing [13741.294984] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13741.306785] Lustre: 375647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13741.564495] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13744.242297] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13744.259659] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13744.509456] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13752.393033] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13766.268648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13766.628717] Lustre: tst250fs-OST0001: new disk, initializing [13766.641232] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13766.652790] Lustre: 376521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13766.872257] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13773.523600] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13773.543210] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13773.690561] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13774.627837] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13781.651824] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13784.474810] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13784.480507] Lustre: Skipped 2 previous similar messages [13804.517818] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13804.531082] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13804.561057] Lustre: Skipped 3 previous similar messages [13804.605556] Lustre: Skipped 3 previous similar messages [13809.638920] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13809.660769] Lustre: Skipped 3 previous similar messages [13810.850103] Lustre: server umount tst250fs-MDT0000 complete [13815.524296] LustreError: 374009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142222 with bad export cookie 2428555993450853779 [13815.525647] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13815.556582] LustreError: 374009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13816.024944] Lustre: server umount tst250fs-MDT0001 complete [13830.691841] Lustre: server umount tst250fs-OST0000 complete [13845.892362] Lustre: server umount tst250fs-OST0001 complete [13866.990601] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [13877.557310] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [13914.437394] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:25:19 (1787142319) [13916.264111] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13933.690513] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [13944.814931] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [13976.777564] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13638 sec ========= 08:26:22 (1787142382) [13978.693252] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:26:23 (1787142383) === [13982.640890] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:26:27 (1787142387) === [14022.321694] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [14024.901644] Key type lgssc unregistered [14025.185708] LNet: 385657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14025.189696] LNetError: 385657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14025.206871] LNet: Removed LNI 192.168.201.136@tcp [14026.163179] Key type .llcrypt unregistered [14026.164868] Key type ._llcrypt unregistered