[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 518669869 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001026] APIC: Switch to symmetric I/O mode setup [ 0.002598] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.007647] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.011016] LSM: Security Framework initializing [ 0.012052] Yama: becoming mindful. [ 0.014022] SELinux: Initializing. [ 0.015083] *** VALIDATE selinux *** [ 0.022491] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027477] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029180] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030180] *** VALIDATE tmpfs *** [ 0.032118] *** VALIDATE proc *** [ 0.033235] *** VALIDATE cgroup *** [ 0.034006] *** VALIDATE cgroup2 *** [ 0.035288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038023] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.041855] debug: unmapping init [mem 0xffffffffa3a59000-0xffffffffa3a60fff] [ 0.043955] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044965] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046029] ... version: 2 [ 0.047035] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049025] ... value mask: 0000ffffffffffff [ 0.050032] ... max period: 00007fffffffffff [ 0.051010] ... fixed-purpose events: 3 [ 0.052019] ... event mask: 000000070000000f [ 0.053465] rcu: Hierarchical SRCU implementation. [ 0.056942] smp: Bringing up secondary CPUs ... [ 0.057596] x86: Booting SMP configuration: [ 0.058016] .... node #0, CPUs: #1 #2 #3 [ 0.064094] smp: Brought up 1 node, 4 CPUs [ 0.066010] smpboot: Max logical packages: 1 [ 0.067020] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.125068] node 0 deferred pages initialised in 56ms [ 0.129116] devtmpfs: initialized [ 0.130402] x86/mm: Memory block size: 128MB [ 0.134419] gcov: version magic: 0x41383552 [ 0.138402] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139300] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140448] pinctrl core: initialized pinctrl subsystem [ 0.141330] [ 0.141899] ************************************************************* [ 0.142009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143009] ** ** [ 0.144008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.145009] ** ** [ 0.146144] ** This means that this kernel is built to expose internal ** [ 0.147006] ** IOMMU data structures, which may compromise security on ** [ 0.148006] ** your system. ** [ 0.149019] ** ** [ 0.150007] ** If you see this message and you are not debugging the ** [ 0.151006] ** kernel, report this immediately to your vendor! ** [ 0.152007] ** ** [ 0.153009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154006] ************************************************************* [ 0.157390] NET: Registered protocol family 16 [ 0.163076] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175097] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.191012] cpuidle: using governor menu [ 0.192951] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198434] PCI: Using configuration type 1 for base access [ 0.203188] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220129] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.226095] cryptd: max_cpu_qlen set to 1000 [ 0.228213] ACPI: Added _OSI(Module Device) [ 0.230009] ACPI: Added _OSI(Processor Device) [ 0.234011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.247578] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255954] ACPI: Interpreter enabled [ 0.258168] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260100] ACPI: Using IOAPIC for interrupt routing [ 0.262195] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266618] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.281638] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.284054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.287026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.291217] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.299100] acpiphp: Slot [2] registered [ 0.301169] acpiphp: Slot [5] registered [ 0.304179] acpiphp: Slot [6] registered [ 0.306430] acpiphp: Slot [7] registered [ 0.308275] acpiphp: Slot [8] registered [ 0.310185] acpiphp: Slot [9] registered [ 0.312092] acpiphp: Slot [10] registered [ 0.313138] acpiphp: Slot [3] registered [ 0.315106] acpiphp: Slot [4] registered [ 0.317110] acpiphp: Slot [11] registered [ 0.319139] acpiphp: Slot [12] registered [ 0.321097] acpiphp: Slot [13] registered [ 0.322155] acpiphp: Slot [14] registered [ 0.324180] acpiphp: Slot [15] registered [ 0.326200] acpiphp: Slot [16] registered [ 0.328165] acpiphp: Slot [17] registered [ 0.329254] acpiphp: Slot [18] registered [ 0.331134] acpiphp: Slot [19] registered [ 0.333099] acpiphp: Slot [20] registered [ 0.335089] acpiphp: Slot [21] registered [ 0.337125] acpiphp: Slot [22] registered [ 0.338316] acpiphp: Slot [23] registered [ 0.340092] acpiphp: Slot [24] registered [ 0.342092] acpiphp: Slot [25] registered [ 0.343103] acpiphp: Slot [26] registered [ 0.345085] acpiphp: Slot [27] registered [ 0.346182] acpiphp: Slot [28] registered [ 0.348105] acpiphp: Slot [29] registered [ 0.350131] acpiphp: Slot [30] registered [ 0.351085] acpiphp: Slot [31] registered [ 0.353056] PCI host bridge to bus 0000:00 [ 0.354013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.357019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.359016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.363026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.365017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.368023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.370423] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373790] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.377548] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.388916] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.393054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.395016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.397015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.400014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.404116] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.406937] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.409043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.413012] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.418017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.434071] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.440018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.447012] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.456026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.462034] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.477021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.486767] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.492018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.499015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.509016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.512000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.512000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.512000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.522019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.531087] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.537023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.541023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.553020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.563556] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.569020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.574017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.588019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.598000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.607017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.617018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.633023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.648521] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.650313] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.653558] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.656457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.658224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.663089] iommu: Default domain type: Passthrough [ 0.664766] SCSI subsystem initialized [ 0.665000] ACPI: bus type USB registered [ 0.665000] usbcore: registered new interface driver usbfs [ 0.665000] usbcore: registered new interface driver hub [ 0.665000] usbcore: registered new device driver usb [ 0.665000] pps_core: LinuxPPS API ver. 1 registered [ 0.665000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.667065] PTP clock support registered [ 0.670145] EDAC MC: Ver: 3.0.0 [ 0.672158] PCI: Using ACPI for IRQ routing [ 0.673854] NetLabel: Initializing [ 0.675013] NetLabel: domain hash size = 128 [ 0.676008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.677097] NetLabel: unlabeled traffic allowed by default [ 0.679166] vgaarb: loaded [ 0.681334] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.683014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.690000] clocksource: Switched to clocksource kvm-clock [ 0.829448] VFS: Disk quotas dquot_6.6.0 [ 0.831095] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.833758] *** VALIDATE ramfs *** [ 0.835139] *** VALIDATE hugetlbfs *** [ 0.836457] pnp: PnP ACPI init [ 0.838931] pnp: PnP ACPI: found 6 devices [ 0.855928] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.858614] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.860540] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.862589] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.864637] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.866719] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.869744] NET: Registered protocol family 2 [ 0.872298] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.878130] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.881056] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.885807] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.889714] TCP: Hash tables configured (established 65536 bind 65536) [ 0.893383] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.896416] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.899170] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.901803] NET: Registered protocol family 1 [ 0.904558] RPC: Registered named UNIX socket transport module. [ 0.907023] RPC: Registered udp transport module. [ 0.908810] RPC: Registered tcp transport module. [ 0.910544] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.912517] NET: Registered protocol family 44 [ 0.914311] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.916601] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.919046] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.921230] PCI: CLS 0 bytes, default 64 [ 0.923954] Unpacking initramfs... [ 2.741824] debug: unmapping init [mem 0xffff8cd03cc54000-0xffff8cd03ffbffff] [ 2.748177] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.752191] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.757084] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.917396] Initialise system trusted keyrings [ 3.919172] Key type blacklist registered [ 3.921036] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.931572] zbud: loaded [ 3.934582] *** VALIDATE nfs *** [ 3.935488] *** VALIDATE nfs4 *** [ 3.940297] pstore: using deflate compression [ 3.944360] Platform Keyring initialized [ 4.098682] NET: Registered protocol family 38 [ 4.102488] Key type asymmetric registered [ 4.105699] Asymmetric key parser 'x509' registered [ 4.108232] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.111637] io scheduler mq-deadline registered [ 4.113751] io scheduler kyber registered [ 4.115749] io scheduler bfq registered [ 4.117645] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.120849] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.123092] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.127198] ACPI: Power Button [PWRF] [ 4.141185] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.149166] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.173378] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.188321] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.215972] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.248750] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.279435] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.284511] Non-volatile memory driver v1.3 [ 4.286300] Linux agpgart interface v0.103 [ 4.318414] virtio_blk virtio1: [vda] 134664 512-byte logical blocks (68.9 MB/65.8 MiB) [ 4.321747] vda: detected capacity change from 0 to 68947968 [ 4.375374] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.378680] vdb: detected capacity change from 0 to 1073741824 [ 4.395307] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.398684] vdc: detected capacity change from 0 to 2621440000 [ 4.422536] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.427020] vdd: detected capacity change from 0 to 2621440000 [ 4.444404] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.448462] vde: detected capacity change from 0 to 4294967296 [ 4.470575] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.474102] vdf: detected capacity change from 0 to 4294967296 [ 4.495476] libphy: Fixed MDIO Bus: probed [ 4.506588] usbcore: registered new interface driver usbserial_generic [ 4.509655] usbserial: USB Serial support registered for generic [ 4.512855] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.516709] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.519222] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.522253] mousedev: PS/2 mouse device common for all mice [ 4.525065] rtc_cmos 00:05: RTC can wake from S4 [ 4.529949] rtc_cmos 00:05: registered as rtc0 [ 4.534678] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.538212] intel_pstate: CPU model not supported [ 4.540308] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.548280] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.552689] hid: raw HID events driver (C) Jiri Kosina [ 4.555240] usbcore: registered new interface driver usbhid [ 4.557634] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.559840] usbhid: USB HID core driver [ 4.565323] drop_monitor: Initializing network drop monitor service [ 4.567721] Initializing XFRM netlink socket [ 4.570435] NET: Registered protocol family 10 [ 4.574238] Segment Routing with IPv6 [ 4.575969] NET: Registered protocol family 17 [ 4.578043] mpls_gso: MPLS GSO support [ 4.584500] RAS: Correctable Errors collector initialized. [ 4.587316] AVX version of gcm_enc/dec engaged. [ 4.588913] AES CTR mode by8 optimization enabled [ 4.690899] sched_clock: Marking stable (4690740805, 0)->(5883980526, -1193239721) [ 4.695353] registered taskstats version 1 [ 4.697729] Loading compiled-in X.509 certificates [ 4.701346] zswap: loaded using pool lzo/zbud [ 4.738992] Key type big_key registered [ 4.759413] Key type encrypted registered [ 4.761413] ima: No TPM chip found, activating TPM-bypass! [ 4.763561] ima: Allocated hash algorithm: sha1 [ 4.765977] ima: No architecture policies found [ 4.767713] evm: Initialising EVM extended attributes: [ 4.769647] evm: security.selinux [ 4.770611] evm: security.ima [ 4.771743] evm: security.capability [ 4.773927] evm: HMAC attrs: 0x1 [ 4.777125] rtc_cmos 00:05: setting system clock to 2026-03-07 17:52:46 UTC (1772905966) [ 4.786152] debug: unmapping init [mem 0xffffffffa4a03000-0xffffffffa4bfffff] [ 4.789838] debug: unmapping init [mem 0xffffffffa3782000-0xffffffffa3a58fff] [ 4.796264] Write protecting the kernel read-only data: 28672k [ 4.799681] debug: unmapping init [mem 0xffffffffa1e03000-0xffffffffa1ffffff] [ 4.802322] debug: unmapping init [mem 0xffffffffa2714000-0xffffffffa27fffff] [ 4.842605] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.853343] systemd[1]: Detected virtualization kvm. [ 4.855964] systemd[1]: Detected architecture x86-64. [ 4.858461] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.888376] systemd[1]: No hostname configured. [ 4.890529] systemd[1]: Set hostname to . [ 4.893380] random: systemd: uninitialized urandom read (16 bytes read) [ 4.896168] systemd[1]: Initializing machine ID from random generator. [ 5.011247] random: ln: uninitialized urandom read (6 bytes read) [ 5.200982] random: systemd: uninitialized urandom read (16 bytes read) [ 5.204090] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.211891] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.222198] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.424148] device-mapper: uevent: version 1.0.3 [ 6.430298] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 8.163206] random: fast init done [ 8.217778] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.353105] scsi host0: ata_piix [ 8.475179] scsi host1: ata_piix [ 8.477274] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.480268] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.281185] random: crng init done [ 13.282613] random: 7 urandom warning(s) missed due to ratelimiting [ 15.140239] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.263075] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.200440] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.643712] SELinux: Disabled at runtime. [ 18.723367] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.732792] systemd[1]: Detected virtualization kvm. [ 18.736614] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.783782] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.790417] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.798299] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.802947] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.806262] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.818329] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.827792] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch[ 20.136314] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.950935] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.971551] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.452869] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.661803] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.713952] EDAC sbridge: Ver: 1.1.2 [ 26.316521] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 27.065722] NFS: Registering the id_resolver key type [ 27.067313] Key type id_resolver registered [ 27.070539] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg436-server login: [ 52.760527] libcfs: loading out-of-tree module taints kernel. [ 52.780803] Key type ._llcrypt registered [ 52.782411] Key type .llcrypt registered [ 52.821249] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 59.709239] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 60.244112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.249413] alg: No test for adler32 (adler32-zlib) [ 61.188307] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 61.422085] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 63.015105] Key type lgssc registered [ 63.544080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 69.788142] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 84.264855] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 88.748633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 88.757986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 89.842953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 89.856117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 89.894717] Lustre: lustre-MDT0000: new disk, initializing [ 89.917231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 89.922875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 91.199967] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.438037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 96.476300] Lustre: 6475:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 96.489903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 96.492341] Lustre: Skipped 1 previous similar message [ 96.528827] Lustre: lustre-MDT0001: new disk, initializing [ 96.549872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 96.558613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 96.562818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 97.881639] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 100.083485] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 103.232606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 103.326726] Lustre: lustre-OST0000: new disk, initializing [ 103.328600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 103.343415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 105.236189] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 108.015693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 108.018553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 108.041072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 110.642934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 110.690166] Lustre: lustre-OST0001: new disk, initializing [ 110.692086] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 110.709462] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 112.455962] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 114.476222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 114.478578] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 114.492510] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 118.270862] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 122.966544] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 129.867072] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing check_logdir /tmp/testlogs/ [ 131.332478] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing yml_node [ 133.014376] Lustre: DEBUG MARKER: Client: 2.17.50.193 [ 134.004759] Lustre: DEBUG MARKER: MDS: 2.17.50.193 [ 134.939919] Lustre: DEBUG MARKER: OSS: 2.17.50.193 [ 135.566270] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Mar 7 12:54:56 EST 2026 [ 142.232315] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 142.947731] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 155.616942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 155.617299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 155.623061] Lustre: Skipped 3 previous similar messages [ 155.625083] Lustre: Skipped 2 previous similar messages [ 160.736226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 160.738408] Lustre: Skipped 4 previous similar messages [ 161.442552] Lustre: server umount lustre-MDT0000 complete [ 163.013899] LustreError: 9433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906124 with bad export cookie 17405251819570573236 [ 163.015445] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 163.018637] LustreError: 9433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 165.855941] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 165.856366] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 165.860045] Lustre: Skipped 1 previous similar message [ 165.864236] Lustre: Skipped 1 previous similar message [ 168.864114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 168.866817] Lustre: Skipped 1 previous similar message [ 169.187356] Lustre: server umount lustre-MDT0001 complete [ 170.626513] Lustre: server umount lustre-OST0000 complete [ 172.146086] Lustre: server umount lustre-OST0001 complete [ 174.635756] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 178.200539] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 192.913050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.019909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.033587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.074236] Lustre: lustre-MDT0000: new disk, initializing [ 193.106461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.115114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.598531] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 199.456386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.506630] Lustre: 14936:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.626131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.070856] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 204.838984] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 206.462950] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 209.245090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.371603] Lustre: lustre-OST0000: new disk, initializing [ 209.373159] Lustre: Skipped 1 previous similar message [ 209.375226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.378568] Lustre: Skipped 2 previous similar messages [ 209.414990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.418060] Lustre: Skipped 1 previous similar message [ 211.068916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.072040] Lustre: Skipped 1 previous similar message [ 211.073846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.116076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.639395] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 214.855476] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 216.368614] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 216.450657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 217.987454] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 218.077111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 220.888494] Lustre: server umount lustre-OST0000 complete [ 225.247864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 225.250995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 225.256908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 231.931424] Lustre: server umount lustre-MDT0000 complete [ 233.239352] LustreError: 14929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906194 with bad export cookie 17405251819570574636 [ 233.242698] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 233.243090] LustreError: 14929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 242.702335] Lustre: server umount lustre-MDT0001 complete [ 244.631318] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:56:45 (1772906205) [ 251.165188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 251.334409] LustreError: 19400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 251.341033] LustreError: 19400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 251.358573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.831986] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 256.012894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 257.627808] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 259.884148] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 261.450778] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 262.324273] Lustre: Failing over lustre-MDT0000 [ 262.410769] Lustre: server umount lustre-MDT0000 complete [ 265.376692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.377649] LustreError: 19426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.382551] Lustre: Skipped 2 previous similar messages [ 270.816108] LustreError: 19419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 270.822249] LustreError: 19419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 275.770563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 275.824450] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 277.441279] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 281.059502] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 281.062317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 281.070120] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 281.082292] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 286.179133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 286.180925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.181994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.182026] Lustre: Skipped 4 previous similar messages [ 286.190331] Lustre: Skipped 1 previous similar message [ 289.370585] Lustre: server umount lustre-MDT0000 complete [ 290.827074] LustreError: 21720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906252 with bad export cookie 17405251819570575868 [ 290.830832] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 290.831593] LustreError: 21720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 290.947280] Lustre: server umount lustre-MDT0001 complete [ 294.225159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 294.423056] LustreError: 22394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.453916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 294.456785] Lustre: Skipped 2 previous similar messages [ 295.894399] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 299.268106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 300.862268] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 302.995025] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 304.564974] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 305.365689] Lustre: Failing over lustre-MDT0000 [ 305.455274] Lustre: server umount lustre-MDT0000 complete [ 309.472672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.478653] LustreError: 22390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.485960] LustreError: 22390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 318.727853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 318.789932] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.378967] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 323.729532] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 324.065417] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 324.068933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 324.071513] Lustre: Skipped 1 previous similar message [ 324.078333] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 329.183960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 329.186446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.189604] Lustre: Skipped 1 previous similar message [ 329.191224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.193211] Lustre: Skipped 1 previous similar message [ 331.934285] Lustre: server umount lustre-MDT0000 complete [ 333.401855] LustreError: 23112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906295 with bad export cookie 17405251819570576673 [ 333.405856] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 333.408032] LustreError: 23112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 333.522076] Lustre: server umount lustre-MDT0001 complete [ 336.698530] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:58:17 (1772906297) [ 342.438616] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 345.977107] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 361.626447] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 365.636359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.743966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 365.757058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 365.798521] Lustre: lustre-MDT0000: new disk, initializing [ 365.841960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 365.845090] Lustre: Skipped 2 previous similar messages [ 365.853957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 367.277812] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 371.906571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.951106] Lustre: 28368:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 372.055036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 373.458796] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 376.841614] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 378.152351] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 380.517620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 381.827101] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 381.857121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 381.866055] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 382.614314] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 386.010501] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 388.482855] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 395.378367] Lustre: server umount lustre-OST0000 complete [ 397.280721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.281622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 397.285392] Lustre: Skipped 2 previous similar messages [ 397.288691] Lustre: Skipped 2 previous similar messages [ 411.103106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 411.174159] Lustre: server umount lustre-MDT0000 complete [ 412.411315] LustreError: 28359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906374 with bad export cookie 17405251819570577030 [ 412.414742] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 412.416605] LustreError: 28359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 422.933438] Lustre: server umount lustre-MDT0001 complete [ 426.629925] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 427.632359] Key type lgssc unregistered [ 427.761287] LNet: 31933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.763707] LNetError: 31933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 427.772273] LNet: Removed LNI 192.168.204.136@tcp [ 428.109113] Key type .llcrypt unregistered [ 428.110573] Key type ._llcrypt unregistered [ 437.428592] Key type ._llcrypt registered [ 437.430161] Key type .llcrypt registered [ 437.472824] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 437.898684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 437.904273] alg: No test for adler32 (adler32-zlib) [ 438.764295] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 438.858186] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 440.439126] Key type lgssc registered [ 440.781235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 443.269971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 443.276676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.431060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.726596] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 448.629353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.748750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 449.955057] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 451.617086] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 452.790404] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 455.033528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.166838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.030483] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 458.662486] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 458.765239] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 464.457202] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:00:25 (1772906425) [ 465.124511] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 465.741475] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 467.922490] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:00:28 (1772906428) [ 470.684429] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:00:31 (1772906431) [ 471.043106] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 473.896639] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 473.899682] Lustre: Skipped 2 previous similar messages [ 474.464382] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:00:35 (1772906435) [ 481.256831] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 481.259225] Lustre: Skipped 10 previous similar messages [ 482.090617] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:00:42 (1772906442) [ 482.980600] Lustre: *** cfs_fail_loc=131b, val=2*** [ 484.150619] Lustre: *** cfs_fail_loc=131b, val=2*** [ 486.767815] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:00:47 (1772906447) [ 490.999929] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 491.002662] Lustre: Skipped 7 previous similar messages [ 491.523929] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:00:52 (1772906452) [ 496.607711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.609922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.613472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.682044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.686761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 499.689870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.802389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 504.804298] Lustre: Skipped 1 previous similar message [ 509.407133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 509.484238] Lustre: server umount lustre-OST0000 complete [ 515.040500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.040935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 515.043294] Lustre: Skipped 1 previous similar message [ 515.044638] Lustre: Skipped 1 previous similar message [ 520.159897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 520.161372] Lustre: Skipped 2 previous similar messages [ 525.279147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 525.335588] Lustre: server umount lustre-MDT0000 complete [ 526.480077] LustreError: 34919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906488 with bad export cookie 1497762374396788281 [ 526.481885] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 526.482785] LustreError: 34919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 526.592630] Lustre: server umount lustre-MDT0001 complete [ 529.932915] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 530.914371] Key type lgssc unregistered [ 531.035370] LNet: 40785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 531.037456] LNetError: 40785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 531.048301] LNet: Removed LNI 192.168.204.136@tcp [ 531.341325] Key type .llcrypt unregistered [ 531.342321] Key type ._llcrypt unregistered [ 536.866163] Key type ._llcrypt registered [ 536.867144] Key type .llcrypt registered [ 536.903202] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 542.918086] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 543.339611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 543.351788] alg: No test for adler32 (adler32-zlib) [ 544.192296] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 544.277312] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 545.855132] Key type lgssc registered [ 546.193764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 557.620320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 557.625657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.720448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 558.732549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 558.769577] Lustre: lustre-MDT0000: new disk, initializing [ 558.794589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.802856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 560.088833] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 564.328684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.369879] Lustre: 43875:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 564.392669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 564.394395] Lustre: Skipped 1 previous similar message [ 564.429193] Lustre: lustre-MDT0001: new disk, initializing [ 564.464327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 564.476432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 564.479952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 565.705395] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 568.692467] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 569.875394] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 572.058244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.155910] Lustre: lustre-OST0000: new disk, initializing [ 572.158496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 572.187566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.174889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 573.178514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 573.207692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 573.974811] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 576.978658] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 578.253584] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 578.314234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 579.508800] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 579.574609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 583.648599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.649398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.651352] LustreError: Skipped 1 previous similar message [ 583.655915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 583.658427] Lustre: Skipped 1 previous similar message [ 586.257516] Lustre: server umount lustre-OST0000 complete [ 588.768598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.769237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.773859] Lustre: Skipped 2 previous similar messages [ 593.697099] Lustre: server umount lustre-MDT0000 complete [ 594.871170] LustreError: 44655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906556 with bad export cookie 5509706423730291867 [ 594.873532] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.874381] LustreError: 44655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 594.980988] Lustre: server umount lustre-MDT0001 complete [ 597.897192] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:02:38 (1772906558) [ 599.981835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.123045] LustreError: 47925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.129898] LustreError: 47925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 600.145665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.370194] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 604.103595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.208873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 605.366819] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 607.014027] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 608.170970] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 610.370860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 612.271538] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 613.945231] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 631.250071] LustreError: 50138:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 631.546631] Lustre: Modifying parameter general.timeout=1 in log params [ 632.687620] Lustre: Modifying parameter general.timeout=5 in log params [ 632.689239] Lustre: Skipped 3 previous similar messages [ 634.930603] Lustre: Modifying parameter general.timeout=13 in log params [ 634.932552] Lustre: Skipped 7 previous similar messages [ 639.040396] Lustre: Modifying parameter general.timeout=27 in log params [ 639.042483] Lustre: Skipped 13 previous similar messages [ 647.163527] Lustre: Modifying parameter general.timeout=56 in log params [ 647.165356] Lustre: Skipped 28 previous similar messages [ 663.287319] Lustre: Modifying parameter general.timeout=114 in log params [ 663.288676] Lustre: Skipped 57 previous similar messages [ 670.899987] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:03:51 (1772906631) [ 671.806633] Lustre: 57036:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 672.923230] Lustre: 57232:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 672.926958] Lustre: 57232:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 682.812534] Lustre: 58871:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 682.815366] Lustre: 58871:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 3 previous similar messages [ 682.820120] LustreError: 58871:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 683.950022] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:04:04 (1772906644) [ 689.294093] Lustre: server umount lustre-MDT0000 complete [ 690.430157] LustreError: 47907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906652 with bad export cookie 5509706423730292637 [ 690.434457] LustreError: 47907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 690.435306] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 690.656184] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.656639] LustreError: 47951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.660249] Lustre: Skipped 1 previous similar message [ 690.660758] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 690.667727] Lustre: Skipped 1 previous similar message [ 692.573277] Lustre: server umount lustre-MDT0001 complete [ 693.758675] Lustre: server umount lustre-OST0000 complete [ 701.685746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.769896] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 701.778216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 701.780950] Lustre: Skipped 38 previous similar messages [ 701.853260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.856013] Lustre: Skipped 1 previous similar message [ 703.013480] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 705.674741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 705.704707] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 705.714293] Lustre: 61392:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 707.112779] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 708.946042] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 710.142915] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 712.249586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 712.313815] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 714.233411] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 716.054827] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 721.958845] Lustre: 63618:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 727.522264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 727.522434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 727.522955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 727.527635] Lustre: Skipped 1 previous similar message [ 729.810034] Lustre: server umount lustre-OST0000 complete [ 731.615674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 731.617601] LustreError: Skipped 1 previous similar message [ 731.618639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.622249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 731.624259] Lustre: Skipped 1 previous similar message [ 737.041935] Lustre: server umount lustre-MDT0000 complete [ 737.760277] LustreError: 61403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 737.764042] LustreError: 61403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 738.208697] LustreError: 62118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906699 with bad export cookie 5509706423730294779 [ 738.211429] LustreError: 62118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 738.212294] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 738.318179] Lustre: server umount lustre-MDT0001 complete [ 741.689411] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 742.646845] Key type lgssc unregistered [ 742.780379] LNet: 64804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.782631] LNetError: 64804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.791257] LNet: Removed LNI 192.168.204.136@tcp [ 743.067744] Key type .llcrypt unregistered [ 743.068591] Key type ._llcrypt unregistered [ 751.437185] Key type ._llcrypt registered [ 751.438075] Key type .llcrypt registered [ 751.467877] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 751.723793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 751.729908] alg: No test for adler32 (adler32-zlib) [ 752.565815] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 752.642962] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 754.215128] Key type lgssc registered [ 754.544900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 756.876452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 756.881096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.020770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 759.105659] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 761.710257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.876864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 763.151804] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 764.856718] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 765.969528] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 767.963990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 768.125216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.811596] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 771.397319] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 789.910975] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 792.543659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 792.545433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.548552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 793.057783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 793.058088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 793.060814] Lustre: Skipped 1 previous similar message [ 793.062210] Lustre: Skipped 1 previous similar message [ 798.177996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 798.180427] Lustre: Skipped 3 previous similar messages [ 798.479993] Lustre: server umount lustre-MDT0000 complete [ 799.637121] LustreError: 66156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906761 with bad export cookie 10456851961228488342 [ 799.639091] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 799.640416] LustreError: 66156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 799.742497] Lustre: server umount lustre-MDT0001 complete [ 820.739489] Lustre: server umount lustre-OST0000 complete [ 828.670198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.752752] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 828.761048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 828.838197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.967362] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 832.577193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.605800] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 832.614315] Lustre: 70730:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 833.975804] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 835.632208] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 836.730783] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 838.782975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 838.838429] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 838.948776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 838.950777] Lustre: Skipped 1 previous similar message [ 840.586398] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 842.149523] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 847.914356] Lustre: 72962:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 848.536328] Lustre: Modifying parameter general.jobid_var=disable in log params [ 851.424350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 851.426722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.427265] LustreError: Skipped 1 previous similar message [ 851.427635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 851.430383] Lustre: Skipped 1 previous similar message [ 851.435707] Lustre: Skipped 1 previous similar message [ 855.762977] Lustre: server umount lustre-OST0000 complete [ 858.591767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 858.594679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.598805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 866.783993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 866.786007] Lustre: Skipped 3 previous similar messages [ 871.391120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 871.448479] Lustre: server umount lustre-MDT0000 complete [ 871.904218] LustreError: 70741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 871.910409] LustreError: 70741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 872.667032] LustreError: 72367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906834 with bad export cookie 10456851961228489042 [ 872.669365] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 872.669788] LustreError: 72367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 872.780918] Lustre: server umount lustre-MDT0001 complete [ 876.220742] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 877.136638] Key type lgssc unregistered [ 877.263344] LNet: 74153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.265582] LNetError: 74153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.275518] LNet: Removed LNI 192.168.204.136@tcp [ 877.555267] Key type .llcrypt unregistered [ 877.556853] Key type ._llcrypt unregistered [ 886.122638] Key type ._llcrypt registered [ 886.124284] Key type .llcrypt registered [ 886.157466] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 886.479722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.509697] alg: No test for adler32 (adler32-zlib) [ 887.360852] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 887.442541] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 889.015189] Key type lgssc registered [ 889.363257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 891.802949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 891.809752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.963066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 894.181446] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 896.852030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 897.030948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 898.285624] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 899.983779] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 901.155608] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 903.377343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 903.565634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.468794] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 907.127817] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 924.933634] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 925.508609] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 925.510433] Lustre: Skipped 1 previous similar message [ 926.663615] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 926.665305] Lustre: Skipped 3 previous similar messages [ 928.683756] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 928.686265] Lustre: Skipped 6 previous similar messages [ 932.896426] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 932.898222] Lustre: Skipped 14 previous similar messages [ 941.055601] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 941.057434] Lustre: Skipped 28 previous similar messages [ 955.973538] Lustre: 83286:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 956.804805] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:08:37 (1772906917) [ 957.316768] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 957.873818] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:08:38 (1772906918) [ 958.944394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.944604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.945469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.949283] Lustre: Skipped 1 previous similar message [ 964.066062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.068356] Lustre: Skipped 2 previous similar messages [ 969.184866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.186896] Lustre: Skipped 1 previous similar message [ 973.279095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 973.329045] Lustre: server umount lustre-OST0000 complete [ 978.911718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 978.913714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.917316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 980.565094] Lustre: server umount lustre-MDT0000 complete [ 981.715097] LustreError: 75504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772906943 with bad export cookie 3923724086702850935 [ 981.717385] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 981.718393] LustreError: 75504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 981.833649] Lustre: server umount lustre-MDT0001 complete [ 985.095250] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 986.032202] Key type lgssc unregistered [ 986.155309] LNet: 84517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.157793] LNetError: 84517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.169282] LNet: Removed LNI 192.168.204.136@tcp [ 986.428058] Key type .llcrypt unregistered [ 986.428853] Key type ._llcrypt unregistered [ 987.744607] Key type ._llcrypt registered [ 987.745463] Key type .llcrypt registered [ 987.776530] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 988.927272] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules [ 989.238101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 989.278344] alg: No test for adler32 (adler32-zlib) [ 990.131213] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 990.217692] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 991.791165] Key type lgssc registered [ 992.146345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 992.485033] LustreError: 85638:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 999.789171] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1003.150210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1013.804677] LustreError: 85638:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1013.814062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.041546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.443742] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1019.377904] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:09:40 (1772906980) [ 1020.967876] LustreError: 86676:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1020.971325] LustreError: 86676:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1020.974191] LustreError: 86676:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1021.099183] Lustre: server umount lustre-MDT0000 complete [ 1025.612976] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1026.579881] Key type lgssc unregistered [ 1026.706544] LNet: 88131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.708609] LNetError: 88131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.716303] LNet: Removed LNI 192.168.204.136@tcp [ 1027.000361] Key type .llcrypt unregistered [ 1027.001386] Key type ._llcrypt unregistered [ 1033.553618] Key type ._llcrypt registered [ 1033.555286] Key type .llcrypt registered [ 1033.591582] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1033.967782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.973567] alg: No test for adler32 (adler32-zlib) [ 1034.817391] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 1034.892755] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1036.471142] Key type lgssc registered [ 1036.810699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1039.279298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1039.284608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.437408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.612182] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1044.255774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.493108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1045.788718] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1047.526836] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1048.648310] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1050.843236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1051.047713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.929220] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1054.696905] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1091.112528] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:10:51 (1772907051) [ 1091.734714] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1092.368402] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:10:53 (1772907053) [ 1095.647738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1095.650282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.655649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.162901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.164107] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.545101] Lustre: server umount lustre-MDT0000 complete [ 1100.740239] LustreError: 89290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772907062 with bad export cookie 13315732469562491816 [ 1100.742447] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1100.743506] LustreError: 89290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1100.859807] Lustre: server umount lustre-MDT0001 complete [ 1112.377932] Lustre: server umount lustre-OST0000 complete [ 1116.119585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.267525] LustreError: 93261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.274897] LustreError: 93261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1116.316720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.612324] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1120.451495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.700613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1122.017111] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1123.732855] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1124.908214] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1129.087982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1129.171179] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1129.173562] LustreError: 93249:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1129.183118] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1129.185883] LustreError: 95116:0:(tgt_mount.c:2571:server_fill_super()) Unable to start targets: -98 [ 1129.187956] LustreError: 95116:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 1129.189711] LustreError: 95116:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1129.213619] Lustre: server umount lustre-OST0000 complete [ 1129.214922] LustreError: 95116:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1132.049337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1132.111647] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1132.114373] LustreError: 93249:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1132.117496] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1132.120106] LustreError: 95578:0:(tgt_mount.c:2571:server_fill_super()) Unable to start targets: -98 [ 1132.121977] LustreError: 95578:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 1132.123814] LustreError: 95578:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1132.145570] Lustre: server umount lustre-OST0000 complete [ 1132.147905] LustreError: 95578:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1135.154097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1135.218094] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1135.219752] Lustre: Found index 0 for lustre-OST0000, updating log [ 1135.221597] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1135.242327] Lustre: lustre-OST0000: new disk, initializing [ 1135.244455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1135.334141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.163870] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1138.342638] Lustre: Failing over lustre-OST0000 [ 1138.390292] Lustre: server umount lustre-OST0000 complete [ 1139.583164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.584111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.586930] Lustre: Skipped 2 previous similar messages [ 1139.589606] Lustre: Skipped 2 previous similar messages [ 1144.800330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.802996] Lustre: Skipped 1 previous similar message [ 1145.749566] Lustre: server umount lustre-MDT0000 complete [ 1145.824151] LustreError: 96851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1145.828664] LustreError: 96851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1146.960122] LustreError: 93243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772907108 with bad export cookie 13315732469562494448 [ 1146.963153] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1146.963323] LustreError: 93243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1147.078864] Lustre: server umount lustre-MDT0001 complete [ 1149.766531] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:11:50 (1772907110) [ 1154.816296] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1158.175208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.315059] LustreError: 98417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.343936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.589107] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1162.342848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.937030] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1164.940808] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1167.030025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1169.096671] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1171.832928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1171.864197] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1171.897201] Lustre: lustre-OST0001: new disk, initializing [ 1171.898877] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1173.824599] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1176.823430] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1178.116473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1178.119755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1178.173305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1182.640611] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1190.091208] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 1192.927893] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1192.930069] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1195.225485] Lustre: server umount lustre-MDT0001 complete [ 1198.528232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.563981] LustreError: 98414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.569905] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1198.572321] LustreError: 98414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1198.574703] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1198.592246] Lustre: 98405:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1198.872512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1198.874582] Lustre: Skipped 3 previous similar messages [ 1199.783796] LustreError: 102318:0:(obd_mount.c:213:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1199.788675] LustreError: 102318:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1199.791656] LustreError: 98409:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1199.822724] LustreError: 102337:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.204.136@tcp: cfg command failed: rc = -17 [ 1199.825532] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1199.825532] [ 1200.236870] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1203.265341] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 13:12:44 (1772907164) [ 1204.198996] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1208.455561] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1214.185788] Lustre: 104242:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1222.161337] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1223.317888] Lustre: 105376:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1275.872671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1275.872772] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.876329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1275.879747] Lustre: Skipped 4 previous similar messages [ 1275.884010] Lustre: Skipped 8 previous similar messages [ 1280.230437] Lustre: server umount lustre-MDT0000 complete [ 1280.992404] LustreError: 100061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1280.998065] LustreError: 100061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1281.471703] LustreError: 98397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772907243 with bad export cookie 13315732469562495176 [ 1281.472886] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1281.477297] LustreError: 98397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1281.614844] Lustre: server umount lustre-MDT0001 complete [ 1293.331290] Lustre: server umount lustre-OST0000 complete [ 1304.862081] Lustre: server umount lustre-OST0001 complete [ 1306.764537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1309.941093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1312.745282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1315.889300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1322.256364] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1325.924421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.934061] Lustre: lustre-MDT0000: reset Object Index mappings [ 1326.105412] LustreError: 109282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.137806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.477525] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1330.278898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.289857] Lustre: lustre-MDT0001: reset Object Index mappings [ 1331.888403] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1332.858688] Lustre: 110396:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1335.088316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.359330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1337.293674] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1339.900757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1341.161370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1341.162073] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1341.189068] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1341.796845] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1344.838202] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1346.082077] Lustre: 112247:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1366.210254] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:15:27 (1772907327) [ 1397.215778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1397.219722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.223603] Lustre: Skipped 1 previous similar message [ 1397.226221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.388175] Lustre: server umount lustre-MDT0000 complete [ 1402.574170] LustreError: 110779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772907364 with bad export cookie 13315732469562541761 [ 1402.578030] LustreError: 110779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1402.578357] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1402.848139] LustreError: 109279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.852749] LustreError: 109279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1408.957829] Lustre: server umount lustre-MDT0001 complete [ 1416.319494] Lustre: server umount lustre-OST0000 complete [ 1423.705576] Lustre: server umount lustre-OST0001 complete [ 1425.693269] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 1428.957259] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1440.729425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.819989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1440.831279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1440.868418] Lustre: lustre-MDT0000: new disk, initializing [ 1440.898376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1442.233758] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1445.212760] Lustre: server umount lustre-MDT0000 complete [ 1448.239411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.335854] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1448.340390] Lustre: 116451:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1448.345476] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1449.676919] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1451.285781] Lustre: server umount lustre-MDT0000 complete [ 1453.923174] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:16:54 (1772907414) [ 1454.484395] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1455.088835] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:16:55 (1772907415) [ 1460.265893] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1463.547205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.690830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.693047] Lustre: Skipped 5 previous similar messages [ 1464.971416] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1467.705082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.744302] Lustre: 118465:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1467.765418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1467.767202] Lustre: Skipped 1 previous similar message [ 1467.806040] Lustre: lustre-MDT0001: new disk, initializing [ 1467.854301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1467.858089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1469.156780] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1471.411944] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1473.512084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1473.610324] Lustre: lustre-OST0000: new disk, initializing [ 1473.612038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1473.755225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1473.759125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1473.792348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1475.418798] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1479.495434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1480.638918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1480.641758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1480.676332] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1481.483909] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1485.810207] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1486.935091] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1607.679575] Lustre: 127213:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1607.683965] Lustre: 127213:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1607.686244] Lustre: 127213:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1607.688765] Lustre: 127213:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1607.690863] Lustre: 127213:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1607.693541] Lustre: 127213:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1692.579861] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:20:53 (1772907653) [ 1705.955963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1705.957936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1705.960136] Lustre: Skipped 8 previous similar messages [ 1705.963839] Lustre: Skipped 10 previous similar messages [ 1706.321241] Lustre: server umount lustre-MDT0000 complete [ 1707.524336] LustreError: 127238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772907669 with bad export cookie 13315732469562595066 [ 1707.527612] LustreError: 127238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1707.529252] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1711.072452] LustreError: 121649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1711.077873] LustreError: 121649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1713.759805] Lustre: server umount lustre-MDT0001 complete [ 1721.091792] Lustre: server umount lustre-OST0000 complete [ 1728.485543] Lustre: server umount lustre-OST0001 complete [ 1730.634880] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 1734.168678] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1746.171551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.276120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1746.289899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1746.292428] Lustre: Skipped 1 previous similar message [ 1746.333896] Lustre: lustre-MDT0000: new disk, initializing [ 1746.335591] Lustre: Skipped 1 previous similar message [ 1746.361929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.365026] Lustre: Skipped 3 previous similar messages [ 1746.370679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1747.661196] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1751.776031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.815878] Lustre: 130195:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1751.919809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1753.204994] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1756.242437] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1757.382722] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1759.659173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1760.955547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1760.991526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1761.694822] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1764.775446] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1767.777109] Lustre: lustre-MDD0000: changelog on [ 1768.656769] Lustre: lustre-MDD0001: changelog on [ 2113.757449] Lustre: lustre-MDD0001: changelog off [ 2114.378795] Lustre: 130203:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2114.696703] Lustre: lustre-MDD0000: changelog off [ 2119.654094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.654546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2119.659473] Lustre: Skipped 2 previous similar messages [ 2119.665551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2119.667883] Lustre: Skipped 5 previous similar messages [ 2122.453990] Lustre: server umount lustre-OST0000 complete [ 2138.079132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2138.191493] Lustre: server umount lustre-MDT0000 complete [ 2139.445243] LustreError: 130187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908101 with bad export cookie 13315732469563399597 [ 2139.448616] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.448725] LustreError: 130187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2139.577253] Lustre: server umount lustre-MDT0001 complete [ 2143.345046] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2144.395440] Key type lgssc unregistered [ 2144.515372] LNet: 140309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.517725] LNetError: 140309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.524285] LNet: Removed LNI 192.168.204.136@tcp [ 2144.856123] Key type .llcrypt unregistered [ 2144.857320] Key type ._llcrypt unregistered [ 2151.324526] Key type ._llcrypt registered [ 2151.325740] Key type .llcrypt registered [ 2151.366628] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2157.590567] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2157.897663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.906126] alg: No test for adler32 (adler32-zlib) [ 2158.777662] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 2158.876809] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2160.463133] Key type lgssc registered [ 2160.916264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2172.624582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2172.632294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.735168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2173.747283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2173.787248] Lustre: lustre-MDT0000: new disk, initializing [ 2173.813863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2173.821210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2175.128752] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2179.289199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.330474] Lustre: 143446:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2179.353178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2179.355103] Lustre: Skipped 1 previous similar message [ 2179.393448] Lustre: lustre-MDT0001: new disk, initializing [ 2179.425322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2179.437774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2179.441750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2180.776531] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2183.996016] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2185.229304] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2187.629688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2187.743532] Lustre: lustre-OST0000: new disk, initializing [ 2187.745678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2187.775843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.677657] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2192.810923] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2196.473982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2196.477836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2196.512172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2200.311129] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_module obdecho/obdecho [ 2200.942643] LustreError: 146063:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2200.946749] LustreError: 146063:0:(obd_config.c:880:class_setup()) setup ec failed (-22) [ 2201.593276] LustreError: 146159:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2205.151960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2205.154344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.158458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.688451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.689896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.694879] Lustre: Skipped 1 previous similar message [ 2211.162719] Lustre: server umount lustre-MDT0000 complete [ 2212.380378] LustreError: 143440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908174 with bad export cookie 3650847162177029641 [ 2212.385165] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2212.502663] Lustre: server umount lustre-MDT0001 complete [ 2223.960785] Lustre: server umount lustre-OST0000 complete [ 2231.057570] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2234.595775] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2246.695955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2246.809263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2246.825324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2246.868600] Lustre: lustre-MDT0000: new disk, initializing [ 2246.899717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.907750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2248.278446] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2252.395161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2252.448055] Lustre: 149834:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2252.476448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2252.478076] Lustre: Skipped 1 previous similar message [ 2252.513858] Lustre: lustre-MDT0001: new disk, initializing [ 2252.548968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2252.562543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2252.565735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2253.840114] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2257.036350] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2258.250519] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2260.607546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.731608] Lustre: lustre-OST0000: new disk, initializing [ 2260.733718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2260.769712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2262.208743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2262.213087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2262.251134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2262.826116] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2265.861395] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2267.170963] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2267.246538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2268.513040] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2268.584598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2272.743629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.743927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2272.748434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2272.749150] LustreError: Skipped 1 previous similar message [ 2272.757110] Lustre: Skipped 3 previous similar messages [ 2275.478502] Lustre: server umount lustre-OST0000 complete [ 2277.856978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.861691] Lustre: Skipped 1 previous similar message [ 2277.863957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.865691] Lustre: Skipped 1 previous similar message [ 2282.976489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2282.978635] Lustre: Skipped 1 previous similar message [ 2283.169693] Lustre: server umount lustre-MDT0000 complete [ 2284.419560] LustreError: 149825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908246 with bad export cookie 3650847162177030474 [ 2284.426690] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2284.426916] LustreError: 149825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2284.584132] Lustre: server umount lustre-MDT0001 complete [ 2285.819051] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 13:30:46 (1772908246) [ 2286.301895] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2286.887749] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 13:30:47 (1772908247) [ 2289.299794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.463040] LustreError: 153832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.469602] LustreError: 153832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2289.487185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2290.718242] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2293.424142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.963201] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2296.717951] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2297.988244] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2300.274951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2302.389741] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2304.129307] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2324.448324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2324.451158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.455377] Lustre: Skipped 1 previous similar message [ 2324.458288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2327.606143] Lustre: server umount lustre-MDT0000 complete [ 2328.826290] LustreError: 153813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908290 with bad export cookie 3650847162177031244 [ 2328.829766] LustreError: 153813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2328.830042] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2329.797579] LustreError: 153828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.087759] LustreError: 155985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.092105] LustreError: 155985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2338.272203] LustreError: 155347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.278652] LustreError: 155347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2343.391188] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2343.490654] Lustre: server umount lustre-MDT0001 complete [ 2345.596667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2349.844809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.930228] LustreError: 157359:0:(llog.c:1643:llog_backup()) MGC192.168.204.136@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2349.932601] Lustre: 157359:0:(mgc_request_server.c:764:mgc_llog_local_copy()) MGC192.168.204.136@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2370.016451] LustreError: 157364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.021137] LustreError: 157364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2380.255801] LustreError: 157365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.261232] LustreError: 157365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2396.357060] LustreError: 157365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.366861] LustreError: 157365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 2410.975344] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2410.977423] Lustre: 157359:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.136@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2429.407917] LustreError: 157383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.411970] LustreError: 157383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 2441.183316] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2441.185123] Lustre: 153825:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.136@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2451.496735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2451.498826] Lustre: Skipped 2 previous similar messages [ 2452.794261] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2454.499358] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2455.616876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.663244] LustreError: 153825:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 2474.982272] LustreError: 141674:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002f9a1cae x1859028904942720/t0(0) o250->MGC192.168.204.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2475.077215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 2476.535478] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2478.324449] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2478.882885] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2480.127879] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2480.617131] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2482.742281] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2489.376599] Lustre: server umount lustre-MDT0000 complete [ 2490.711622] LustreError: 153813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908452 with bad export cookie 3650847162177037579 [ 2490.716755] LustreError: 153813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2490.717902] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2490.842517] Lustre: server umount lustre-MDT0001 complete [ 2511.906119] Lustre: server umount lustre-OST0000 complete [ 2518.449744] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2521.857924] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2533.752646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.852869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2533.864454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2533.902198] Lustre: lustre-MDT0000: new disk, initializing [ 2533.928939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.930856] Lustre: Skipped 1 previous similar message [ 2533.936598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2535.252050] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2539.302256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.342977] Lustre: 162176:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2539.454480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2540.792605] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2543.810905] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2544.950939] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2547.137795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2549.089982] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2549.945199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2549.985648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2552.061616] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2553.363764] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2553.440634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2554.816919] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2554.920488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2560.480834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2560.481748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2560.485162] LustreError: Skipped 1 previous similar message [ 2560.488294] Lustre: Skipped 3 previous similar messages [ 2560.488782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2560.492602] Lustre: Skipped 11 previous similar messages [ 2561.677602] Lustre: server umount lustre-OST0000 complete [ 2577.375083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2577.429383] Lustre: server umount lustre-MDT0000 complete [ 2578.669522] LustreError: 162168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908540 with bad export cookie 3650847162177054358 [ 2578.674453] LustreError: 162168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2578.674882] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2578.802393] Lustre: server umount lustre-MDT0001 complete [ 2581.608592] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 13:35:42 (1772908542) [ 2583.728212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.888402] LustreError: 166179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.893853] LustreError: 166179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 2585.199493] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2587.929071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.286515] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2591.044529] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2592.155896] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2594.321461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2596.302577] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2598.029832] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2617.251993] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 13:36:18 (1772908578) [ 2618.335780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2618.337921] LustreError: Skipped 1 previous similar message [ 2618.339113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.342507] Lustre: Skipped 3 previous similar messages [ 2618.344120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2618.345670] Lustre: Skipped 6 previous similar messages [ 2624.334703] Lustre: server umount lustre-OST0000 complete [ 2631.699832] Lustre: server umount lustre-MDT0000 complete [ 2632.896102] LustreError: 166161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908594 with bad export cookie 3650847162177055128 [ 2632.898431] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2632.901705] LustreError: 166161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2633.018083] Lustre: server umount lustre-MDT0001 complete [ 2636.529771] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2637.572665] Key type lgssc unregistered [ 2637.709336] LNet: 169570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2637.713181] LNetError: 169570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2637.721427] LNet: Removed LNI 192.168.204.136@tcp [ 2638.039220] Key type .llcrypt unregistered [ 2638.040673] Key type ._llcrypt unregistered [ 2645.430036] Key type ._llcrypt registered [ 2645.430952] Key type .llcrypt registered [ 2645.472404] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2645.876149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2645.909323] alg: No test for adler32 (adler32-zlib) [ 2646.771967] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 2646.863766] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2648.447175] Key type lgssc registered [ 2648.839292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2651.470900] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2651.476767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2666.911373] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2672.031200] LustreError: 170774:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.136@tcp: failed processing log, type 4: rc = -110 [ 2677.151392] LustreError: 170774:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2677.154802] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2677.156535] LustreError: MGC192.168.204.136@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2677.159501] LustreError: 170774:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2677.161884] LustreError: 170774:0:(tgt_mount.c:2571:server_fill_super()) Unable to start targets: -22 [ 2677.164103] LustreError: 170774:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2677.187168] Lustre: server umount lustre-OST0000 complete [ 2677.188578] LustreError: 170774:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2680.003223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.167865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.513767] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2685.452403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.597698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.055634] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2688.840851] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2690.046165] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2692.344181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2692.430246] LustreError: 172861:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2692.433387] Lustre: 172861:0:(mgc_request_server.c:744:mgc_llog_local_copy()) MGC192.168.204.136@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2692.485462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2694.289161] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2696.115969] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2700.769953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.771440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2700.773494] Lustre: Skipped 2 previous similar messages [ 2700.775141] Lustre: Skipped 1 previous similar message [ 2703.003315] Lustre: server umount lustre-MDT0000 complete [ 2704.213197] LustreError: 171255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908665 with bad export cookie 6341950416855344204 [ 2704.216494] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.217243] LustreError: 171255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2704.337605] Lustre: server umount lustre-MDT0001 complete [ 2715.929939] Lustre: server umount lustre-OST0000 complete [ 2718.847635] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2722.027815] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2733.883853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.979888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2733.993221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2734.034070] Lustre: lustre-MDT0000: new disk, initializing [ 2734.059414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.065859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2735.329390] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2739.412773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.458769] Lustre: 176231:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2739.482396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2739.484088] Lustre: Skipped 1 previous similar message [ 2739.522241] Lustre: lustre-MDT0001: new disk, initializing [ 2739.553588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2739.566457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2739.569458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2740.842570] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2743.838095] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2744.974771] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2747.219825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2747.332829] Lustre: lustre-OST0000: new disk, initializing [ 2747.335584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2748.026472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2748.029607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2748.066743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2749.313651] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2752.363944] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2753.681723] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2753.764536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2755.206364] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2755.296869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2758.625181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2758.625828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.627811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2758.631016] Lustre: Skipped 1 previous similar message [ 2758.632659] Lustre: Skipped 1 previous similar message [ 2762.189096] Lustre: server umount lustre-OST0000 complete [ 2763.744870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.745466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.748589] Lustre: Skipped 1 previous similar message [ 2768.864846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2768.868508] Lustre: Skipped 2 previous similar messages [ 2769.596967] Lustre: server umount lustre-MDT0000 complete [ 2770.852128] LustreError: 176225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908732 with bad export cookie 6341950416855344792 [ 2770.856453] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.858020] LustreError: 176225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2770.981828] Lustre: server umount lustre-MDT0001 complete [ 2774.126713] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 13:38:54 (1772908734) [ 2780.469229] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2781.460354] Key type lgssc unregistered [ 2781.588538] LNet: 180746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2781.591720] LNetError: 180746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2781.600310] LNet: Removed LNI 192.168.204.136@tcp [ 2782.040460] Key type .llcrypt unregistered [ 2782.042558] Key type ._llcrypt unregistered [ 2790.872617] Key type ._llcrypt registered [ 2790.873922] Key type .llcrypt registered [ 2790.914626] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2791.233631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2791.289116] alg: No test for adler32 (adler32-zlib) [ 2792.142195] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 2792.228915] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2793.807198] Key type lgssc registered [ 2794.451124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2798.371644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2798.384333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.577300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.140135] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2805.390607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.557524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2807.473652] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2809.678392] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2811.449271] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2814.725168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2814.873032] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2814.876589] LustreError: 183561:0:(llog.c:1684:llog_backup()) MGC192.168.204.136@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2814.882716] Lustre: 183561:0:(mgc_request_server.c:741:mgc_llog_local_copy()) MGC192.168.204.136@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2814.959755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.667300] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2819.848472] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2821.091123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.093238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.099025] Lustre: Skipped 2 previous similar messages [ 2821.099447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2821.104519] Lustre: Skipped 2 previous similar messages [ 2826.211932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.216167] Lustre: Skipped 2 previous similar messages [ 2827.000291] Lustre: server umount lustre-MDT0000 complete [ 2828.786061] LustreError: 181907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908790 with bad export cookie 4933645280845301994 [ 2828.793555] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2828.794328] LustreError: 181907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2828.972591] Lustre: server umount lustre-MDT0001 complete [ 2841.173914] Lustre: server umount lustre-OST0000 complete [ 2844.248687] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2847.543880] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2858.890835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.983030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2858.992749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2859.029542] Lustre: lustre-MDT0000: new disk, initializing [ 2859.051497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.057938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2860.270585] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2864.168650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.208126] Lustre: 186935:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2864.225680] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2864.227439] Lustre: Skipped 1 previous similar message [ 2864.229319] LustreError: 187701:0:(llog.c:1684:llog_backup()) MGC192.168.204.136@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2864.233313] LustreError: 187701:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 2864.236242] Lustre: 187701:0:(mgc_request_server.c:764:mgc_llog_local_copy()) MGC192.168.204.136@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2864.245522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2864.248058] Lustre: Skipped 1 previous similar message [ 2864.285865] Lustre: lustre-MDT0001: new disk, initializing [ 2864.316033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2864.325906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2864.328691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2865.475818] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2868.370677] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2869.445611] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2871.525845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2871.622880] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2871.624098] Lustre: Skipped 1 previous similar message [ 2871.625401] LustreError: 188654:0:(llog.c:1684:llog_backup()) MGC192.168.204.136@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2871.627981] LustreError: 188654:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 2871.630217] Lustre: 188654:0:(mgc_request_server.c:764:mgc_llog_local_copy()) MGC192.168.204.136@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2871.633084] Lustre: 188654:0:(mgc_request_server.c:764:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2871.641166] Lustre: lustre-OST0000: new disk, initializing [ 2871.642825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2873.441627] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2876.463651] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2877.883090] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2879.485291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2879.490399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2879.522324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2880.013102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2881.350732] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2881.420430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2884.579406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2884.580302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.583825] LustreError: Skipped 1 previous similar message [ 2884.591406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2884.596504] Lustre: Skipped 1 previous similar message [ 2888.165422] Lustre: server umount lustre-OST0000 complete [ 2890.208626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2890.212316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2890.219039] Lustre: Skipped 1 previous similar message [ 2890.221911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.817511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.818696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.031180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2904.108038] Lustre: server umount lustre-MDT0000 complete [ 2905.056488] LustreError: 186947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.065791] LustreError: 186947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2905.408421] LustreError: 188677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908867 with bad export cookie 4933645280845302610 [ 2905.412188] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2905.415029] LustreError: 188677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2905.566547] Lustre: server umount lustre-MDT0001 complete [ 2908.725199] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 13:41:09 (1772908869) [ 2914.724257] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2918.484478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.617295] LustreError: 191650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.630699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.634013] Lustre: Skipped 1 previous similar message [ 2919.987518] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2923.945209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2924.000770] LustreError: 191651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.712745] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2927.257808] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2929.634032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2932.209671] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2935.516201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2935.574620] Lustre: lustre-OST0001: new disk, initializing [ 2935.576398] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2935.606320] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2935.608113] Lustre: Skipped 2 previous similar messages [ 2937.870842] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2943.361330] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2944.530714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2944.539127] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2944.594300] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2950.304302] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2955.842408] Lustre: DEBUG MARKER: ADD OST3 [ 2958.103327] loop: module loaded [ 2958.124717] loop0: detected capacity change from 0 to 204800000 [ 2958.224160] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2958.227216] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2958.230827] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2960.765407] loop0: detected capacity change from 0 to 204800000 [ 2962.913128] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2963.008412] Lustre: lustre-OST0003: new disk, initializing [ 2963.010676] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 2963.225051] Lustre: DEBUG MARKER: STOP OST3 [ 2963.973989] Lustre: Failing over lustre-OST0003 [ 2964.016579] Lustre: server umount lustre-OST0003 complete [ 2964.703817] LustreError: 193103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2965.358070] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2969.032460] Lustre: DEBUG MARKER: START OST3 again [ 2969.989479] LustreError: 193104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.997093] LustreError: 193104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2971.498195] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.594244] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 2971.596400] Lustre: Skipped 1 previous similar message [ 2973.057311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 2973.063434] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 2973.078603] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 2973.770628] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2987.348932] Lustre: Failing over lustre-OST0003 [ 2987.398413] Lustre: server umount lustre-OST0003 complete [ 2988.512504] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2988.512668] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.515750] LustreError: Skipped 1 previous similar message [ 2988.516327] LustreError: 196046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.520827] Lustre: Skipped 1 previous similar message [ 2988.532785] LustreError: 196046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3000.801109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3000.805136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.812873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3000.815461] Lustre: Skipped 3 previous similar messages [ 3006.390820] Lustre: server umount lustre-MDT0000 complete [ 3007.458558] LustreError: 191651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.464521] LustreError: 191651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3007.884185] LustreError: 191632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772908969 with bad export cookie 4933645280845303380 [ 3007.891385] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3017.696498] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3017.698287] Lustre: Skipped 7 previous similar messages [ 3022.303129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3022.392302] Lustre: server umount lustre-MDT0001 complete [ 3023.690282] Lustre: server umount lustre-OST0000 complete [ 3024.966842] Lustre: server umount lustre-OST0001 complete [ 3027.087706] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 3029.793189] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3043.264431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3043.376548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3043.386386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3043.425732] Lustre: lustre-MDT0000: new disk, initializing [ 3043.447947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.454833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3044.793545] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3049.268569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.306926] Lustre: 200975:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3049.402726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3050.611135] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3053.848465] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3055.042652] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3057.316218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3059.014302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3059.074743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3059.586580] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3062.665746] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3063.956635] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3064.022928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3065.262906] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3065.324658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3069.410492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.410810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3069.415324] Lustre: Skipped 5 previous similar messages [ 3069.418356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3069.420760] Lustre: Skipped 1 previous similar message [ 3080.159101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3080.208430] Lustre: server umount lustre-OST0000 complete [ 3087.569192] Lustre: server umount lustre-MDT0000 complete [ 3089.108133] LustreError: 202714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772909050 with bad export cookie 4933645280845305473 [ 3089.112099] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.112929] LustreError: 202714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3089.228282] Lustre: server umount lustre-MDT0001 complete [ 3090.839266] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:44:11 (1772909051) [ 3096.362402] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 3099.842555] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3113.335541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.432661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3113.446913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3113.449123] Lustre: Skipped 3 previous similar messages [ 3113.490712] Lustre: lustre-MDT0000: new disk, initializing [ 3113.492063] Lustre: Skipped 2 previous similar messages [ 3113.520978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3113.526708] Lustre: Skipped 2 previous similar messages [ 3114.806651] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3119.153655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.193538] Lustre: 207146:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3119.290963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3120.524728] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3123.838226] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3125.267006] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3128.393609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3129.888377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3130.503873] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3133.863441] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3135.410230] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3135.528672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3136.988144] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3137.088204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3140.065657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3140.066839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.067799] LustreError: Skipped 2 previous similar messages [ 3140.071135] Lustre: Skipped 2 previous similar messages [ 3140.071846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.076771] Lustre: Skipped 7 previous similar messages [ 3152.351155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3152.417461] Lustre: server umount lustre-OST0000 complete [ 3168.223252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3168.343251] Lustre: server umount lustre-MDT0000 complete [ 3170.224382] LustreError: 207140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772909131 with bad export cookie 4933645280845306257 [ 3170.229458] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3170.235395] LustreError: 207140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3170.425438] Lustre: server umount lustre-MDT0001 complete [ 3174.855503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.123263] LustreError: 210956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.130527] LustreError: 210956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3175.167650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.171228] Lustre: Skipped 5 previous similar messages [ 3176.762159] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3180.543244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.172046] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3184.202159] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3185.419650] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3188.373408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3190.838519] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3193.399240] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3211.232626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3211.237604] LustreError: Skipped 1 previous similar message [ 3211.240529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3211.248602] Lustre: Skipped 3 previous similar messages [ 3216.668971] Lustre: server umount lustre-OST0000 complete [ 3224.484066] Lustre: server umount lustre-MDT0000 complete [ 3225.682275] LustreError: 213520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772909187 with bad export cookie 4933645280845307041 [ 3225.685976] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3225.687617] LustreError: 213520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3225.811253] Lustre: server umount lustre-MDT0001 complete [ 3326.275718] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3327.835340] Key type lgssc unregistered [ 3327.988616] LNet: 214287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3327.993502] LNetError: 214287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3328.003619] LNet: Removed LNI 192.168.204.136@tcp [ 3328.489210] Key type .llcrypt unregistered [ 3328.491346] Key type ._llcrypt unregistered [ 3339.803975] Key type ._llcrypt registered [ 3339.805118] Key type .llcrypt registered [ 3339.844581] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 3347.355814] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3347.591214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3347.620343] alg: No test for adler32 (adler32-zlib) [ 3348.554512] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 3348.682273] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3350.295200] Key type lgssc registered [ 3350.975096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3368.203968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3368.214594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.341752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3369.356912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3369.395968] Lustre: lustre-MDT0000: new disk, initializing [ 3369.428037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3369.436889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3371.070898] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3376.612902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.675557] Lustre: 217581:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3376.708607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3376.711056] Lustre: Skipped 1 previous similar message [ 3376.754778] Lustre: lustre-MDT0001: new disk, initializing [ 3376.789876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3376.804665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3376.808532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3378.410919] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3382.330284] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3383.805823] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3386.813217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3386.935494] Lustre: lustre-OST0000: new disk, initializing [ 3386.938714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3386.979326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.173921] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3390.852944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3390.858288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3390.902146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3392.607540] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3394.197733] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3394.275381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3395.741405] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3395.807252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3401.189295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.189775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.191986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.195292] LustreError: Skipped 1 previous similar message [ 3401.203377] Lustre: Skipped 1 previous similar message [ 3402.795631] Lustre: server umount lustre-OST0000 complete [ 3406.305157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.311063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3406.313403] Lustre: Skipped 2 previous similar messages [ 3410.874434] Lustre: server umount lustre-MDT0000 complete [ 3412.675933] LustreError: 217574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772909374 with bad export cookie 3162651286785505159 [ 3412.683295] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3412.683975] LustreError: 217574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3412.844942] Lustre: server umount lustre-MDT0001 complete [ 3421.483355] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3426.361679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.557704] LustreError: 222090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.564161] LustreError: 222090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3426.576411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.203995] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3431.904810] LustreError: 222091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.063621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.239722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3433.855625] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3435.147768] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3438.192736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.622757] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3443.474745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3443.538049] Lustre: lustre-OST0001: new disk, initializing [ 3443.539720] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3443.570527] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3443.573253] Lustre: Skipped 1 previous similar message [ 3445.361041] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3449.340351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3449.343043] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3449.373796] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3454.914045] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3457.428452] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3466.120129] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:50:26 (1772909426) [ 3468.255957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3468.258037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.261298] Lustre: Skipped 1 previous similar message [ 3468.262869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3471.840571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3471.841246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3471.846806] Lustre: Skipped 2 previous similar messages [ 3472.921948] Lustre: server umount lustre-MDT0000 complete [ 3474.093241] LustreError: 224350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772909435 with bad export cookie 3162651286785505943 [ 3474.096501] LustreError: 224350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3474.097389] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3474.211616] Lustre: server umount lustre-MDT0001 complete [ 3486.216155] Lustre: server umount lustre-OST0000 complete [ 3498.083986] Lustre: server umount lustre-OST0001 complete [ 3500.766281] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 3504.484318] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3520.047092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.141660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3520.154280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3520.194505] Lustre: lustre-MDT0000: new disk, initializing [ 3520.219745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.228683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3521.541762] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3525.338221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.374843] Lustre: 228239:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3525.468795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3526.600863] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3529.455502] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3530.515364] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3532.608548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3532.704092] Lustre: lustre-OST0000: new disk, initializing [ 3532.705389] Lustre: Skipped 1 previous similar message [ 3532.706919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3532.708459] Lustre: Skipped 2 previous similar messages [ 3534.417759] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3534.646179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3534.648504] Lustre: Skipped 1 previous similar message [ 3534.650106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3534.676246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3537.421462] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3538.664974] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3538.725271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3539.893187] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3539.954787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3542.668534] Lustre: server umount lustre-OST0000 complete [ 3545.057066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.057533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3545.059905] Lustre: Skipped 1 previous similar message [ 3545.061369] Lustre: Skipped 3 previous similar messages [ 3555.296106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3555.297640] Lustre: Skipped 3 previous similar messages [ 3558.367121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3558.457169] Lustre: server umount lustre-MDT0000 complete [ 3560.172363] LustreError: 228230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772909521 with bad export cookie 3162651286785507945 [ 3560.175261] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.176028] LustreError: 228230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3569.712717] Lustre: server umount lustre-MDT0001 complete [ 3572.861758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.067441] LustreError: 232048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3573.105569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.107954] Lustre: Skipped 2 previous similar messages [ 3574.827328] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3577.988359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.616534] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3581.483299] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3583.012735] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3586.035634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3588.012206] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3589.813591] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3736.037598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.039364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3736.040362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.043473] Lustre: Skipped 1 previous similar message [ 3747.295098] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3747.346884] Lustre: server umount lustre-OST0000 complete [ 3754.582379] Lustre: server umount lustre-MDT0000 complete [ 3755.716139] LustreError: 232029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772909717 with bad export cookie 3162651286785508729 [ 3755.719111] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3755.719260] LustreError: 232029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3755.838506] Lustre: server umount lustre-MDT0001 complete [ 3858.642297] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3859.849780] Key type lgssc unregistered [ 3859.975370] LNet: 235305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3859.977466] LNetError: 235305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3859.989333] LNet: Removed LNI 192.168.204.136@tcp [ 3860.324167] Key type .llcrypt unregistered [ 3860.326307] Key type ._llcrypt unregistered [ 3870.623820] Key type ._llcrypt registered [ 3870.624755] Key type .llcrypt registered [ 3870.656911] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 3877.658615] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3877.958295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3877.967188] alg: No test for adler32 (adler32-zlib) [ 3878.806107] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 3878.881077] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3880.455091] Key type lgssc registered [ 3880.756532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3895.782989] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3899.269740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3899.275398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.364696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3900.374554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3900.409803] Lustre: lustre-MDT0000: new disk, initializing [ 3900.433521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.439538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3901.932278] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3907.370243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.430612] Lustre: 239292:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3907.457322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3907.459077] Lustre: Skipped 1 previous similar message [ 3907.500084] Lustre: lustre-MDT0001: new disk, initializing [ 3907.531900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3907.542628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3907.545429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3908.740476] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3911.477529] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3914.734714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3914.890755] Lustre: lustre-OST0000: new disk, initializing [ 3914.893711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3914.937600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3916.866860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3916.870779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3916.916399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3917.127679] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3922.314164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3922.434167] Lustre: lustre-OST0001: new disk, initializing [ 3922.438282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3922.493662] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3925.250114] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3930.895221] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3931.140265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3931.144539] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3931.182184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3935.642929] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3943.391749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3943.394092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3943.398021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.464343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.467634] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.471026] Lustre: Skipped 1 previous similar message [ 3949.472093] Lustre: server umount lustre-MDT0000 complete [ 3950.582161] LustreError: 240071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772909912 with bad export cookie 18290191260438878168 [ 3950.584392] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3950.586910] LustreError: 240071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3950.698483] Lustre: server umount lustre-MDT0001 complete [ 3962.135542] Lustre: server umount lustre-OST0000 complete [ 3974.148868] Lustre: server umount lustre-OST0001 complete [ 3980.382028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3984.342505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3987.372925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3991.121096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3997.732331] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4001.512753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.519129] Lustre: lustre-MDT0000: reset Object Index mappings [ 4001.683201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.872350] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4006.632098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.638418] Lustre: lustre-MDT0001: reset Object Index mappings [ 4008.185141] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4009.159567] Lustre: 247823:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4011.562955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4011.704602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4011.706470] Lustre: Skipped 1 previous similar message [ 4012.713818] LustreError: 248359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.719825] LustreError: 248359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4012.724154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4013.789469] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4016.631636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4017.780843] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4018.711531] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4022.526320] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4024.142826] Lustre: 249668:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4032.480446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4032.485723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4032.494326] Lustre: Skipped 1 previous similar message [ 4032.498093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4032.501914] Lustre: Skipped 3 previous similar messages [ 4036.890497] Lustre: server umount lustre-MDT0000 complete [ 4038.158500] LustreError: 247443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772909999 with bad export cookie 18290191260438884804 [ 4038.163454] LustreError: 247443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4038.166529] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4038.275832] Lustre: server umount lustre-MDT0001 complete [ 4050.202116] Lustre: server umount lustre-OST0000 complete [ 4062.292571] Lustre: server umount lustre-OST0001 complete [ 4069.310131] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 14:00:29 (1772910029) [ 4075.632390] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4078.906794] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4093.777486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.935410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4093.952569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4094.008473] Lustre: lustre-MDT0000: new disk, initializing [ 4094.044141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.048107] Lustre: Skipped 1 previous similar message [ 4094.057148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4095.547026] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4099.904221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.943527] Lustre: 253882:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4100.044610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4101.797392] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4105.833003] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4107.185451] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4110.382806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4110.561760] Lustre: lustre-OST0000: new disk, initializing [ 4110.564665] Lustre: Skipped 1 previous similar message [ 4110.568604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4110.572813] Lustre: Skipped 2 previous similar messages [ 4112.512297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4112.516903] Lustre: Skipped 1 previous similar message [ 4112.520054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4112.555716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4112.826707] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4116.688379] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4118.475078] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4118.571723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4120.156526] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4120.266620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4123.106873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4123.107669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.112056] LustreError: Skipped 1 previous similar message [ 4123.113476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4123.120572] Lustre: Skipped 4 previous similar messages [ 4123.129694] Lustre: Skipped 5 previous similar messages [ 4128.227909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4128.231909] Lustre: Skipped 1 previous similar message [ 4135.391133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4135.454825] Lustre: server umount lustre-OST0000 complete [ 4138.466622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4138.467896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4138.473604] Lustre: Skipped 1 previous similar message [ 4138.477607] Lustre: Skipped 2 previous similar messages [ 4151.263192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4151.359382] Lustre: server umount lustre-MDT0000 complete [ 4153.075976] LustreError: 253876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772910114 with bad export cookie 18290191260438886771 [ 4153.083037] LustreError: 253876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4153.083728] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4153.228213] Lustre: server umount lustre-MDT0001 complete [ 4160.386641] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4164.600069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.748115] LustreError: 258403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.765229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4164.767163] Lustre: Skipped 2 previous similar messages [ 4166.330508] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4169.736170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.191915] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4172.497100] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4175.305464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4178.189405] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4182.151719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.261946] Lustre: lustre-OST0001: new disk, initializing [ 4182.278954] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4183.910936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4183.915474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4183.978765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4184.881137] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4189.117683] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4190.225517] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4196.874086] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4196.900195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4200.418529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.421591] Lustre: Skipped 1 previous similar message [ 4200.421943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4200.424382] Lustre: Skipped 6 previous similar messages [ 4204.708639] Lustre: server umount lustre-MDT0000 complete [ 4205.541923] LustreError: 258399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.546101] LustreError: 258399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4205.927320] LustreError: 259122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772910167 with bad export cookie 18290191260438887555 [ 4205.931573] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4205.932752] LustreError: 259122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4206.044062] Lustre: server umount lustre-MDT0001 complete [ 4217.874066] Lustre: server umount lustre-OST0000 complete [ 4229.966317] Lustre: server umount lustre-OST0001 complete [ 4236.361600] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4240.790697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4240.924489] LustreError: 263676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.944895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4240.946818] Lustre: Skipped 3 previous similar messages [ 4242.856620] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4245.984061] LustreError: 263677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4246.510632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4248.227677] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4249.536192] Lustre: 264783:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4252.243592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4254.734972] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4256.550676] LustreError: 265733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.296981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4260.488612] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4265.834901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4265.843474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4269.573310] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4271.064692] Lustre: 266634:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4278.097673] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4278.105741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4281.682253] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 14:04:02 (1772910242) [ 4296.677352] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4296.681517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4296.683842] Lustre: Skipped 5 previous similar messages [ 4296.690538] Lustre: Skipped 4 previous similar messages [ 4299.718400] Lustre: server umount lustre-MDT0000 complete [ 4301.510582] LustreError: 263658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772910263 with bad export cookie 18290191260438893729 [ 4301.515392] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4301.515534] LustreError: 263658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4301.792970] LustreError: 263672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.801759] LustreError: 263672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4307.748633] Lustre: server umount lustre-MDT0001 complete [ 4315.724649] Lustre: server umount lustre-OST0000 complete [ 4323.303674] Lustre: server umount lustre-OST0001 complete [ 4325.633141] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4328.461528] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4343.142963] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4347.249768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4347.366074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4347.377110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4347.419154] Lustre: lustre-MDT0000: new disk, initializing [ 4347.447101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4348.840415] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4353.617615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.693606] Lustre: 270817:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4353.853710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4355.554828] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4361.199645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4362.673497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4363.292509] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4368.178723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4369.347212] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4370.686400] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4375.012666] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4385.270703] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 14:05:45 (1772910345) [ 4389.858134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4389.862634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.864352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.864367] Lustre: Skipped 4 previous similar messages [ 4389.880183] Lustre: Skipped 4 previous similar messages [ 4392.232435] Lustre: server umount lustre-MDT0000 complete [ 4393.627215] LustreError: 270810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772910355 with bad export cookie 18290191260438901492 [ 4393.633621] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4393.636628] LustreError: 270810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4393.798544] Lustre: server umount lustre-MDT0001 complete [ 4405.646698] Lustre: server umount lustre-OST0000 complete [ 4417.623167] Lustre: server umount lustre-OST0001 complete [ 4423.991798] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4427.066921] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4442.709511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.843672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4442.846853] Lustre: Skipped 2 previous similar messages [ 4442.854437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4442.855954] Lustre: Skipped 4 previous similar messages [ 4442.890817] Lustre: lustre-MDT0000: new disk, initializing [ 4442.892071] Lustre: Skipped 3 previous similar messages [ 4442.913844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4442.915528] Lustre: Skipped 7 previous similar messages [ 4442.920207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4442.922300] Lustre: Skipped 3 previous similar messages [ 4444.278500] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4449.499739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4449.578365] Lustre: 277854:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4449.736298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4449.738843] Lustre: Skipped 2 previous similar messages [ 4451.266961] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4454.639765] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4456.542684] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4459.648480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4461.456326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4462.131507] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4466.264535] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4467.175760] Lustre: Mounted lustre-client [ 4467.560897] LustreError: 280350:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8ccf83b3a800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4467.601425] Lustre: Unmounted lustre-client [ 4471.777206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4471.777565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.782917] LustreError: Skipped 1 previous similar message [ 4471.791277] Lustre: Skipped 1 previous similar message [ 4473.000325] Lustre: server umount lustre-OST0000 complete [ 4480.630948] Lustre: server umount lustre-MDT0000 complete [ 4482.475279] LustreError: 279606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772910444 with bad export cookie 18290191260438902892 [ 4482.482298] LustreError: 279606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4482.482715] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4482.647120] Lustre: server umount lustre-MDT0001 complete [ 4486.693969] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4488.135358] Key type lgssc unregistered [ 4520.927129] LNet: 1 local NIs in recovery (showing 1): 192.168.204.136@tcp [ 4521.055314] LNet: 281626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4521.059290] LNetError: 281626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4596.837479] LNet: Removed LNI 192.168.204.136@tcp [ 4597.227155] Key type .llcrypt unregistered [ 4597.229260] Key type ._llcrypt unregistered [ 4604.279564] Key type ._llcrypt registered [ 4604.280753] Key type .llcrypt registered [ 4604.313732] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4610.640861] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4611.074322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4611.093842] alg: No test for adler32 (adler32-zlib) [ 4611.985535] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 4612.085309] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4613.695202] Key type lgssc registered [ 4614.184996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4626.284408] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 14:09:47 (1772910587) [ 4628.697993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4628.705799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4629.804122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4629.815656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4629.850928] Lustre: lustre-MDT0000: new disk, initializing [ 4629.873162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.878506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4631.046010] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4635.495682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4635.532755] Lustre: 284783:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4635.553195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4635.554631] Lustre: Skipped 1 previous similar message [ 4635.589708] Lustre: lustre-MDT0001: new disk, initializing [ 4635.618498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4635.629861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4635.632662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4637.254601] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4640.700780] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4642.217787] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4645.296696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4645.398043] Lustre: lustre-OST0000: new disk, initializing [ 4645.399937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4645.427488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.485361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4647.489077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4647.521163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4647.580397] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4651.036793] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4657.634931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4657.636311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.639901] LustreError: Skipped 1 previous similar message [ 4657.648653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4657.654300] Lustre: Skipped 1 previous similar message [ 4661.263185] Lustre: server umount lustre-OST0000 complete [ 4666.336491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4666.342163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4666.350213] Lustre: Skipped 1 previous similar message [ 4666.354113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4667.873355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4667.874610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.050301] Lustre: server umount lustre-MDT0000 complete [ 4670.469714] LustreError: 284777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772910632 with bad export cookie 15381535873720451305 [ 4670.475225] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4670.475649] LustreError: 284777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4670.610874] Lustre: server umount lustre-MDT0001 complete [ 4675.216464] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4676.486137] Key type lgssc unregistered [ 4676.623412] LNet: 288153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4676.626268] LNetError: 288153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4676.638333] LNet: Removed LNI 192.168.204.136@tcp [ 4676.988540] Key type .llcrypt unregistered [ 4676.990286] Key type ._llcrypt unregistered [ 4687.282951] Key type ._llcrypt registered [ 4687.283944] Key type .llcrypt registered [ 4687.316852] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4687.620550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4687.662077] alg: No test for adler32 (adler32-zlib) [ 4688.505703] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 4688.580463] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4690.159137] Key type lgssc registered [ 4690.657230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4694.442870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4694.454814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4695.658502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.247304] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4701.167058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4701.315984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4702.772852] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4705.052973] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4706.848294] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4709.799328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4709.925220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4712.200833] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4714.702921] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4728.018695] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4728.505311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4730.580431] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 4731.363285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.366560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4731.373338] Lustre: Skipped 2 previous similar messages [ 4731.378012] Lustre: Skipped 2 previous similar messages [ 4732.066775] Lustre: server umount lustre-MDT0000 complete [ 4733.553209] LustreError: 289511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772910695 with bad export cookie 9435442485538871343 [ 4733.560081] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4733.560421] LustreError: 289511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4735.685755] LustreError: 289526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4735.697772] LustreError: 289526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4735.704354] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 4736.480305] LustreError: 290699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4736.480668] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4738.528190] LustreError: 289527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.528444] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4738.536019] Lustre: Skipped 1 previous similar message [ 4739.808782] Lustre: server umount lustre-MDT0001 complete [ 4745.937798] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 4747.690973] Lustre: server umount lustre-OST0000 complete [ 4751.653872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4753.203143] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4754.594727] Lustre: 293279:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.136@tcp [ 4755.615145] Lustre: server umount MGS complete [ 4759.618541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4759.791569] LustreError: 293850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4759.821899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4761.738512] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4765.152378] LustreError: 293851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4765.782729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.959684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4767.789491] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4771.485442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4774.460449] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4777.045239] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4788.387760] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 4790.043659] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4790.805672] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4795.363418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.365951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4795.372122] Lustre: Skipped 1 previous similar message [ 4795.378171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4798.249541] Lustre: server umount lustre-OST0000 complete [ 4800.481517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.491015] Lustre: Skipped 1 previous similar message [ 4806.267614] Lustre: server umount lustre-MDT0000 complete [ 4807.970564] LustreError: 293832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772910769 with bad export cookie 9435442485538872064 [ 4807.975199] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4807.977660] LustreError: 293832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4808.132587] Lustre: server umount lustre-MDT0001 complete [ 4812.490244] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4813.896499] Key type lgssc unregistered [ 4814.082651] LNet: 297048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4814.088162] LNetError: 297048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4814.098624] LNet: Removed LNI 192.168.204.136@tcp [ 4814.593281] Key type .llcrypt unregistered [ 4814.596083] Key type ._llcrypt unregistered [ 4822.870961] Key type ._llcrypt registered [ 4822.872447] Key type .llcrypt registered [ 4822.926557] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4829.058101] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4829.328264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4829.333882] alg: No test for adler32 (adler32-zlib) [ 4830.180715] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 4830.253825] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4831.823119] Key type lgssc registered [ 4832.129545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4845.147465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4845.156277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.252170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4846.262838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4846.301084] Lustre: lustre-MDT0000: new disk, initializing [ 4846.327198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4846.334646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4847.720452] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4852.139101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4852.180774] Lustre: 300159:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4852.201396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4852.202944] Lustre: Skipped 1 previous similar message [ 4852.238023] Lustre: lustre-MDT0001: new disk, initializing [ 4852.270923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4852.283562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4852.287250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4853.512940] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4856.714347] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4857.984723] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4860.663160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4860.763574] Lustre: lustre-OST0000: new disk, initializing [ 4860.765291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4860.790100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.389832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4862.392621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4862.425340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4862.678723] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4865.966137] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4867.454795] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4867.525789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4869.004408] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4869.078093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4871.897918] Lustre: server umount lustre-OST0000 complete [ 4877.792921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4877.792987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4877.793800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4877.799357] Lustre: Skipped 1 previous similar message [ 4882.913783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4882.921198] Lustre: Skipped 2 previous similar messages [ 4882.962474] Lustre: server umount lustre-MDT0000 complete [ 4884.612453] LustreError: 300151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772910846 with bad export cookie 10452502788978361380 [ 4884.616641] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4884.617118] LustreError: 300151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4895.280367] Lustre: server umount lustre-MDT0001 complete [ 4899.522887] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 14:14:19 (1772910859) [ 4920.086051] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 14:14:40 (1772910880) [ 4936.445434] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4937.422926] Key type lgssc unregistered [ 4937.538342] LNet: 306699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4937.540422] LNetError: 306699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4937.549273] LNet: Removed LNI 192.168.204.136@tcp [ 4937.817856] Key type .llcrypt unregistered [ 4937.818858] Key type ._llcrypt unregistered [ 4967.858295] Key type ._llcrypt registered [ 4967.859306] Key type .llcrypt registered [ 4967.895136] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4967.982441] Key type .llcrypt unregistered [ 4967.983415] Key type ._llcrypt unregistered [ 4981.572572] Key type ._llcrypt registered [ 4981.574030] Key type .llcrypt registered [ 4981.625607] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4981.776536] Key type .llcrypt unregistered [ 4981.778660] Key type ._llcrypt unregistered [ 4996.914297] Key type ._llcrypt registered [ 4996.915341] Key type .llcrypt registered [ 4996.951906] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4997.064434] Key type .llcrypt unregistered [ 4997.065956] Key type ._llcrypt unregistered [ 5028.442828] Key type ._llcrypt registered [ 5028.444650] Key type .llcrypt registered [ 5028.506165] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5028.676372] Key type .llcrypt unregistered [ 5028.678561] Key type ._llcrypt unregistered [ 5040.910282] Key type ._llcrypt registered [ 5040.911280] Key type .llcrypt registered [ 5040.950423] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5041.055493] Key type .llcrypt unregistered [ 5041.057088] Key type ._llcrypt unregistered [ 5051.857567] Key type ._llcrypt registered [ 5051.858682] Key type .llcrypt registered [ 5051.898464] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5052.004293] Key type .llcrypt unregistered [ 5052.005321] Key type ._llcrypt unregistered [ 5064.996996] Key type ._llcrypt registered [ 5064.998241] Key type .llcrypt registered [ 5065.034384] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5065.163561] Key type .llcrypt unregistered [ 5065.164995] Key type ._llcrypt unregistered [ 5077.611602] Key type ._llcrypt registered [ 5077.613030] Key type .llcrypt registered [ 5077.673344] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5077.823296] Key type .llcrypt unregistered [ 5077.825423] Key type ._llcrypt unregistered [ 5090.445790] Key type ._llcrypt registered [ 5090.447218] Key type .llcrypt registered [ 5090.493711] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5090.627338] Key type .llcrypt unregistered [ 5090.628356] Key type ._llcrypt unregistered [ 5112.051360] Key type ._llcrypt registered [ 5112.052229] Key type .llcrypt registered [ 5112.091277] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5112.187253] Key type .llcrypt unregistered [ 5112.188160] Key type ._llcrypt unregistered [ 5123.111447] Key type ._llcrypt registered [ 5123.112626] Key type .llcrypt registered [ 5123.165288] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5123.270171] Key type .llcrypt unregistered [ 5123.271204] Key type ._llcrypt unregistered [ 5134.812784] Key type ._llcrypt registered [ 5134.814399] Key type .llcrypt registered [ 5134.863403] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5134.979463] Key type .llcrypt unregistered [ 5134.981490] Key type ._llcrypt unregistered [ 5156.911973] Key type ._llcrypt registered [ 5156.913837] Key type .llcrypt registered [ 5156.966444] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 5164.800075] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5165.089171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5165.099200] alg: No test for adler32 (adler32-zlib) [ 5165.943510] Lustre: Lustre: Build Version: 2.17.50_193_g952cf82 [ 5166.022785] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 5167.599196] Key type lgssc registered [ 5168.223722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5181.852337] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5046 sec ========== 14:19:02 (1772911142) [ 5182.716832] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:19:03 (1772911143) === [ 5184.214850] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:19:04 (1772911144) === [ 5199.927951] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5201.197930] Key type lgssc unregistered [ 5201.311365] LNet: 325508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5201.313339] LNetError: 325508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5201.322296] LNet: Removed LNI 192.168.204.136@tcp [ 5201.568738] Key type .llcrypt unregistered [ 5201.569712] Key type ._llcrypt unregistered