[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 730783342 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003266] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010145] LSM: Security Framework initializing [ 0.011074] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.020635] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024646] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027122] *** VALIDATE tmpfs *** [ 0.028457] *** VALIDATE proc *** [ 0.029270] *** VALIDATE cgroup *** [ 0.030012] *** VALIDATE cgroup2 *** [ 0.032174] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035032] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039187] debug: unmapping init [mem 0xffffffffa0a59000-0xffffffffa0a60fff] [ 0.041928] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042728] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043026] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045014] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047015] ... max period: 00007fffffffffff [ 0.048018] ... fixed-purpose events: 3 [ 0.049018] ... event mask: 000000070000000f [ 0.050309] rcu: Hierarchical SRCU implementation. [ 0.052522] smp: Bringing up secondary CPUs ... [ 0.053407] x86: Booting SMP configuration: [ 0.054024] .... node #0, CPUs: #1 #2 #3 [ 0.057078] smp: Brought up 1 node, 4 CPUs [ 0.059016] smpboot: Max logical packages: 1 [ 0.060016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133069] node 0 deferred pages initialised in 70ms [ 0.137011] devtmpfs: initialized [ 0.138240] x86/mm: Memory block size: 128MB [ 0.140954] gcov: version magic: 0x41383552 [ 0.142388] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148422] pinctrl core: initialized pinctrl subsystem [ 0.150205] [ 0.150688] ************************************************************* [ 0.152025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154010] ** ** [ 0.157014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159012] ** ** [ 0.161021] ** This means that this kernel is built to expose internal ** [ 0.164020] ** IOMMU data structures, which may compromise security on ** [ 0.167033] ** your system. ** [ 0.170021] ** ** [ 0.173018] ** If you see this message and you are not debugging the ** [ 0.176020] ** kernel, report this immediately to your vendor! ** [ 0.179013] ** ** [ 0.182016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185015] ************************************************************* [ 0.188608] NET: Registered protocol family 16 [ 0.190465] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199114] cpuidle: using governor menu [ 0.201000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203675] PCI: Using configuration type 1 for base access [ 0.206147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224129] cryptd: max_cpu_qlen set to 1000 [ 0.228297] ACPI: Added _OSI(Module Device) [ 0.230081] ACPI: Added _OSI(Processor Device) [ 0.232012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239871] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249359] ACPI: Interpreter enabled [ 0.251086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252020] ACPI: Using IOAPIC for interrupt routing [ 0.253193] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256585] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.280090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287091] acpiphp: Slot [2] registered [ 0.288168] acpiphp: Slot [5] registered [ 0.290154] acpiphp: Slot [6] registered [ 0.291121] acpiphp: Slot [7] registered [ 0.293118] acpiphp: Slot [8] registered [ 0.294106] acpiphp: Slot [9] registered [ 0.296107] acpiphp: Slot [10] registered [ 0.297108] acpiphp: Slot [3] registered [ 0.299130] acpiphp: Slot [4] registered [ 0.300105] acpiphp: Slot [11] registered [ 0.302107] acpiphp: Slot [12] registered [ 0.303106] acpiphp: Slot [13] registered [ 0.305154] acpiphp: Slot [14] registered [ 0.306130] acpiphp: Slot [15] registered [ 0.307131] acpiphp: Slot [16] registered [ 0.309110] acpiphp: Slot [17] registered [ 0.310104] acpiphp: Slot [18] registered [ 0.311142] acpiphp: Slot [19] registered [ 0.313120] acpiphp: Slot [20] registered [ 0.314169] acpiphp: Slot [21] registered [ 0.316128] acpiphp: Slot [22] registered [ 0.317113] acpiphp: Slot [23] registered [ 0.319137] acpiphp: Slot [24] registered [ 0.321161] acpiphp: Slot [25] registered [ 0.322098] acpiphp: Slot [26] registered [ 0.324186] acpiphp: Slot [27] registered [ 0.326159] acpiphp: Slot [28] registered [ 0.327151] acpiphp: Slot [29] registered [ 0.329149] acpiphp: Slot [30] registered [ 0.331180] acpiphp: Slot [31] registered [ 0.333079] PCI host bridge to bus 0000:00 [ 0.334027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.339031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.342030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.344022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348199] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352051] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355356] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364667] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369939] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.380743] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383939] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389787] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395030] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.408030] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.413702] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419668] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.441017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.449018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.472021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.486171] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.511029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.518026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.544021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.551927] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.567019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.577018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.613024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.633547] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.646024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.655027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.672029] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.681318] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.696028] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.711026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.750019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.778884] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.794024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.803020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.831016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.840747] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.842257] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.844264] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.846312] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.847142] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.851145] iommu: Default domain type: Passthrough [ 0.853486] SCSI subsystem initialized [ 0.854136] ACPI: bus type USB registered [ 0.856093] usbcore: registered new interface driver usbfs [ 0.857105] usbcore: registered new interface driver hub [ 0.860104] usbcore: registered new device driver usb [ 0.861179] pps_core: LinuxPPS API ver. 1 registered [ 0.863010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.866126] PTP clock support registered [ 0.868076] EDAC MC: Ver: 3.0.0 [ 0.871129] PCI: Using ACPI for IRQ routing [ 0.873192] NetLabel: Initializing [ 0.874014] NetLabel: domain hash size = 128 [ 0.875011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.876104] NetLabel: unlabeled traffic allowed by default [ 0.878086] vgaarb: loaded [ 0.879269] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.881011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.889000] clocksource: Switched to clocksource kvm-clock [ 0.988946] VFS: Disk quotas dquot_6.6.0 [ 0.990527] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.992982] *** VALIDATE ramfs *** [ 0.994311] *** VALIDATE hugetlbfs *** [ 0.996577] pnp: PnP ACPI init [ 0.998986] pnp: PnP ACPI: found 6 devices [ 1.014270] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.017297] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.019241] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.020687] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.023078] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.025598] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.028614] NET: Registered protocol family 2 [ 1.031443] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.037037] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.040268] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.046376] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.050352] TCP: Hash tables configured (established 65536 bind 65536) [ 1.052593] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.055763] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.057982] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.061157] NET: Registered protocol family 1 [ 1.063755] RPC: Registered named UNIX socket transport module. [ 1.065982] RPC: Registered udp transport module. [ 1.067835] RPC: Registered tcp transport module. [ 1.069607] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.072062] NET: Registered protocol family 44 [ 1.074130] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.076401] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.078497] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.081147] PCI: CLS 0 bytes, default 64 [ 1.083520] Unpacking initramfs... [ 2.549308] debug: unmapping init [mem 0xffff9725bcc54000-0xffff9725bffbffff] [ 2.553350] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.555838] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.559184] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.072486] Initialise system trusted keyrings [ 3.074039] Key type blacklist registered [ 3.075902] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.086278] zbud: loaded [ 3.089262] *** VALIDATE nfs *** [ 3.090716] *** VALIDATE nfs4 *** [ 3.092486] pstore: using deflate compression [ 3.096646] Platform Keyring initialized [ 3.227681] NET: Registered protocol family 38 [ 3.229575] Key type asymmetric registered [ 3.231244] Asymmetric key parser 'x509' registered [ 3.233179] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.236383] io scheduler mq-deadline registered [ 3.238269] io scheduler kyber registered [ 3.239948] io scheduler bfq registered [ 3.241785] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.245227] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.248359] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.251653] ACPI: Power Button [PWRF] [ 3.257898] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.265866] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.277727] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.291301] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.316541] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.346543] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.375949] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.382147] Non-volatile memory driver v1.3 [ 3.383823] Linux agpgart interface v0.103 [ 3.421697] virtio_blk virtio1: [vda] 149792 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.423694] vda: detected capacity change from 0 to 76693504 [ 3.436357] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.438844] vdb: detected capacity change from 0 to 1073741824 [ 3.453291] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.456301] vdc: detected capacity change from 0 to 2621440000 [ 3.472550] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.475479] vdd: detected capacity change from 0 to 2621440000 [ 3.497929] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.501707] vde: detected capacity change from 0 to 4294967296 [ 3.518494] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.521923] vdf: detected capacity change from 0 to 4294967296 [ 3.541883] libphy: Fixed MDIO Bus: probed [ 3.547734] usbcore: registered new interface driver usbserial_generic [ 3.550372] usbserial: USB Serial support registered for generic [ 3.552556] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.557068] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.559281] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.562099] mousedev: PS/2 mouse device common for all mice [ 3.565258] rtc_cmos 00:05: RTC can wake from S4 [ 3.570550] rtc_cmos 00:05: registered as rtc0 [ 3.572655] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.575499] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.575626] intel_pstate: CPU model not supported [ 3.578526] hid: raw HID events driver (C) Jiri Kosina [ 3.584936] usbcore: registered new interface driver usbhid [ 3.586783] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.587912] usbhid: USB HID core driver [ 3.588138] drop_monitor: Initializing network drop monitor service [ 3.594551] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.595636] Initializing XFRM netlink socket [ 3.600432] NET: Registered protocol family 10 [ 3.604166] Segment Routing with IPv6 [ 3.605877] NET: Registered protocol family 17 [ 3.608224] mpls_gso: MPLS GSO support [ 3.614367] RAS: Correctable Errors collector initialized. [ 3.616683] AVX version of gcm_enc/dec engaged. [ 3.617935] AES CTR mode by8 optimization enabled [ 3.685187] sched_clock: Marking stable (3685034046, 0)->(4605479287, -920445241) [ 3.690386] registered taskstats version 1 [ 3.693344] Loading compiled-in X.509 certificates [ 3.696215] zswap: loaded using pool lzo/zbud [ 3.719799] Key type big_key registered [ 3.730232] Key type encrypted registered [ 3.731867] ima: No TPM chip found, activating TPM-bypass! [ 3.733646] ima: Allocated hash algorithm: sha1 [ 3.735452] ima: No architecture policies found [ 3.737298] evm: Initialising EVM extended attributes: [ 3.739205] evm: security.selinux [ 3.740454] evm: security.ima [ 3.741641] evm: security.capability [ 3.742911] evm: HMAC attrs: 0x1 [ 3.745246] rtc_cmos 00:05: setting system clock to 2026-09-05 00:29:44 UTC (1788568184) [ 3.750303] debug: unmapping init [mem 0xffffffffa1a03000-0xffffffffa1bfffff] [ 3.752897] debug: unmapping init [mem 0xffffffffa0782000-0xffffffffa0a58fff] [ 3.763105] Write protecting the kernel read-only data: 28672k [ 3.766326] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 3.768892] debug: unmapping init [mem 0xffffffff9f714000-0xffffffff9f7fffff] [ 3.799836] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.807618] systemd[1]: Detected virtualization kvm. [ 3.809527] systemd[1]: Detected architecture x86-64. [ 3.811216] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.837144] systemd[1]: No hostname configured. [ 3.838796] systemd[1]: Set hostname to . [ 3.840652] random: systemd: uninitialized urandom read (16 bytes read) [ 3.842862] systemd[1]: Initializing machine ID from random generator. [ 3.893824] random: ln: uninitialized urandom read (6 bytes read) [ 3.975804] random: systemd: uninitialized urandom read (16 bytes read) [ 3.978714] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.983424] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.989351] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.644369] device-mapper: uevent: version 1.0.3 [ 4.646498] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.293219] random: fast init done [ 5.295701] virtio_net virtio0 ens2: renamed from eth0 [ 5.339681] scsi host0: ata_piix [ 5.377613] scsi host1: ata_piix [ 5.381205] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.383501] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.266141] random: crng init done [ 10.270298] random: 7 urandom warning(s) missed due to ratelimiting [ 10.779229] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.107333] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.729626] printk: systemd: 25 output lines suppressed due to ratelimiting [ 17.390709] SELinux: Disabled at runtime. [ 17.481147] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.494769] systemd[1]: Detected virtualization kvm. [ 17.497874] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.134978] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.144514] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.174303] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.195724] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.201808] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.230531] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.274910] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [[ 19.832079] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 20.538591] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.418341] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.522311] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.083575] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.200274] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 28.389894] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 29.452516] NFS: Registering the id_resolver key type [ 29.454079] Key type id_resolver registered [ 29.454977] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg252-server login: [ 66.932853] hrtimer: interrupt took 3844885 ns [ 98.655944] libcfs: loading out-of-tree module taints kernel. [ 98.752210] Key type ._llcrypt registered [ 98.755970] Key type .llcrypt registered [ 98.909711] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 118.733805] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 121.016600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.026374] alg: No test for adler32 (adler32-zlib) [ 122.740400] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 124.063558] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 125.975250] Key type lgssc registered [ 128.625906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.270668] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 193.864781] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 209.873482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 209.900818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.313309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 211.349384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 211.508296] Lustre: lustre-MDT0000: new disk, initializing [ 211.656843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.721684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 217.304302] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.524698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.674524] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 233.728683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 233.734736] Lustre: Skipped 1 previous similar message [ 233.937784] Lustre: lustre-MDT0001: new disk, initializing [ 234.033561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 234.068022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 234.092687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 239.688775] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.892251] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 255.989143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.362712] Lustre: lustre-OST0000: new disk, initializing [ 256.370763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.379426] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.474328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 263.720226] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.729243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 263.765213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 263.876801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 280.339922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 280.578207] Lustre: lustre-OST0001: new disk, initializing [ 280.592299] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 280.600477] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 280.670531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 289.025944] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.370853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 289.388921] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 289.511819] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 302.163133] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 313.224503] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 321.973620] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing check_logdir /tmp/testlogs/ [ 328.572683] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing yml_node [ 333.840442] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 336.871553] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 340.105719] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 342.176901] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 20:35:19 EDT 2026 [ 362.324844] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 364.117659] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 381.921871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.924266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 381.938147] Lustre: Skipped 1 previous similar message [ 381.944476] Lustre: Skipped 3 previous similar messages [ 386.669418] Lustre: server umount lustre-MDT0000 complete [ 391.121163] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788568571 with bad export cookie 296865789972480458 [ 391.128970] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 391.138110] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 392.162663] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.166947] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 392.192044] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 392.238522] Lustre: Skipped 2 previous similar messages [ 392.250321] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 396.193569] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.210590] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 396.218187] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 396.233669] Lustre: Skipped 2 previous similar messages [ 398.006844] Lustre: server umount lustre-MDT0001 complete [ 403.918451] Lustre: server umount lustre-OST0000 complete [ 408.501940] Lustre: server umount lustre-OST0001 complete [ 416.988522] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 425.694824] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 466.436713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.727280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 466.787153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 466.862194] Lustre: lustre-MDT0000: new disk, initializing [ 466.962975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.999310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 472.487961] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 483.867678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.994944] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 484.093294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 484.097415] Lustre: Skipped 1 previous similar message [ 484.171920] Lustre: lustre-MDT0001: new disk, initializing [ 484.267534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 484.311516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 484.335213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 489.132897] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 500.433511] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 506.692825] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 513.577668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 513.886913] Lustre: lustre-OST0000: new disk, initializing [ 513.908564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 513.918805] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 514.032229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.407656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 515.427750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 515.638646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 520.603359] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 531.476803] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 537.483534] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 537.816752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 543.173475] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 543.414348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 546.272939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 546.275186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.288021] LustreError: Skipped 1 previous similar message [ 546.310175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 546.315806] Lustre: Skipped 2 previous similar messages [ 546.351213] Lustre: Skipped 1 previous similar message [ 551.412500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 551.429335] Lustre: Skipped 1 previous similar message [ 552.279562] Lustre: server umount lustre-OST0000 complete [ 561.122695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 561.145467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.161535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.062281] Lustre: server umount lustre-MDT0000 complete [ 566.754785] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.788248] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 567.460907] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788568748 with bad export cookie 296865789972481991 [ 567.465117] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 567.478926] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 567.713064] Lustre: server umount lustre-MDT0001 complete [ 573.869948] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:39:11 (1788568751) [ 596.421597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 597.193078] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.285485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.893453] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 602.601808] LustreError: 19931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.758157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.812691] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.841554] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 618.586896] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 627.798269] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 634.924712] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 638.027915] Lustre: Failing over lustre-MDT0000 [ 638.484447] Lustre: server umount lustre-MDT0000 complete [ 642.016662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.021437] LustreError: 19927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.029719] Lustre: Skipped 2 previous similar messages [ 657.375115] Lustre: 3649:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788568822/real 1788568822] req@ffff97263b42d180 x1875449798103040/t0(0) o400->MGC192.168.202.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788568838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 657.408922] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.569526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.618794] LustreError: 3646:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff972607fead80 x1875449798105600/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 667.960440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.970409] Lustre: Skipped 1 previous similar message [ 668.038666] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 673.251234] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 673.278152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 673.305900] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 675.254627] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 687.461438] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 693.734713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 693.746863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.776426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 693.798145] Lustre: Skipped 1 previous similar message [ 698.842377] Lustre: server umount lustre-MDT0000 complete [ 701.920568] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.937967] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 703.920608] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788568884 with bad export cookie 296865789972483405 [ 703.928698] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.936889] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 704.409065] Lustre: server umount lustre-MDT0001 complete [ 715.394200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.973529] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 733.001949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.130735] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 747.845202] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 754.875459] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 757.264753] Lustre: Failing over lustre-MDT0000 [ 757.726707] Lustre: server umount lustre-MDT0000 complete [ 758.311512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.336112] Lustre: Skipped 2 previous similar messages [ 768.480198] LustreError: 23271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.504055] LustreError: 23271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 774.623437] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788568939/real 1788568939] req@ffff972503593100 x1875449798134656/t0(0) o400->MGC192.168.202.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788568955 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 774.665719] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 778.320905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.894454] LustreError: 3646:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff972503590700 x1875449798137856/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 786.442070] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 791.403510] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 791.524533] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 791.543955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 791.565035] Lustre: Skipped 1 previous similar message [ 791.581309] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 802.753395] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 807.318425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.357165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 807.368240] Lustre: Skipped 3 previous similar messages [ 813.982912] Lustre: server umount lustre-MDT0000 complete [ 818.519230] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788568999 with bad export cookie 296865789972484343 [ 818.524982] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 818.538679] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 819.188322] Lustre: server umount lustre-MDT0001 complete [ 829.256912] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:43:27 (1788569007) [ 848.017787] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 861.457929] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 906.569505] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 918.886334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.491618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 919.552401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 919.623762] Lustre: lustre-MDT0000: new disk, initializing [ 919.709569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 919.719098] Lustre: Skipped 3 previous similar messages [ 919.755777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 925.203741] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 936.936893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.146672] Lustre: 29649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 937.570465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 943.348787] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 952.566927] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 959.250833] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 965.695624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 965.900524] Lustre: 31619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 967.364325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 967.391946] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 967.518400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 967.534742] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 972.402141] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 983.397914] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 989.440828] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1013.734662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.737319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1013.757877] Lustre: Skipped 1 previous similar message [ 1013.760548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.760555] Lustre: Skipped 2 previous similar messages [ 1026.015123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1026.254468] Lustre: server umount lustre-OST0000 complete [ 1037.115952] Lustre: server umount lustre-MDT0000 complete [ 1039.328971] LustreError: 29657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.349327] LustreError: 29657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 1042.297529] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569223 with bad export cookie 296865789972484749 [ 1042.299030] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1042.306928] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1042.848774] Lustre: server umount lustre-MDT0001 complete [ 1057.216356] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1061.233513] Key type lgssc unregistered [ 1061.691570] LNet: 33596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.704693] LNetError: 33596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.726865] LNet: Removed LNI 192.168.202.152@tcp [ 1062.992125] Key type .llcrypt unregistered [ 1062.993690] Key type ._llcrypt unregistered [ 1093.290526] Key type ._llcrypt registered [ 1093.292814] Key type .llcrypt registered [ 1093.386215] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1094.269133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1094.398261] alg: No test for adler32 (adler32-zlib) [ 1095.603488] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1095.900460] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1097.647356] Key type lgssc registered [ 1099.106308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1109.093525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1109.171321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.786666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.928980] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1120.292996] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.320787] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1125.344296] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.951372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.434103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1132.016720] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1140.014111] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1146.646908] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1155.016793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1155.480479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.512926] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1171.972326] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1172.227212] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1188.893796] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:49:26 (1788569366) [ 1191.088695] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1193.354311] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1202.018910] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:49:39 (1788569379) [ 1212.472506] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:49:49 (1788569389) [ 1213.533676] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1222.550144] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1222.571507] Lustre: Skipped 2 previous similar messages [ 1224.949781] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:50:02 (1788569402) [ 1227.127793] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1227.136712] Lustre: Skipped 2 previous similar messages [ 1237.088690] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1237.099215] Lustre: Skipped 4 previous similar messages [ 1253.321137] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:50:30 (1788569430) [ 1256.366666] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1259.762652] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1269.498987] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:50:46 (1788569446) [ 1271.496699] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1271.501815] Lustre: Skipped 6 previous similar messages [ 1285.797923] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:51:03 (1788569463) [ 1305.065310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.072700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1305.074147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.088208] Lustre: Skipped 1 previous similar message [ 1305.101552] LustreError: Skipped 1 previous similar message [ 1310.191469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1310.205980] Lustre: Skipped 2 previous similar messages [ 1315.300946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1315.311185] Lustre: Skipped 1 previous similar message [ 1316.320172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1316.655856] Lustre: server umount lustre-OST0000 complete [ 1320.423140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1320.453099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.550459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.560531] Lustre: Skipped 2 previous similar messages [ 1326.985880] Lustre: server umount lustre-MDT0000 complete [ 1330.656395] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.670787] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1331.534885] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569512 with bad export cookie 13087855838397652455 [ 1331.543334] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1331.550727] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1331.961444] Lustre: server umount lustre-MDT0001 complete [ 1344.350499] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1347.711630] Key type lgssc unregistered [ 1348.015577] LNet: 42868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.023994] LNetError: 42868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.093783] LNet: Removed LNI 192.168.202.152@tcp [ 1350.257280] Key type .llcrypt unregistered [ 1350.261667] Key type ._llcrypt unregistered [ 1372.171615] Key type ._llcrypt registered [ 1372.174539] Key type .llcrypt registered [ 1372.362447] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 1389.551474] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1390.715924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1390.968296] alg: No test for adler32 (adler32-zlib) [ 1392.209550] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1392.636930] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1394.383178] Key type lgssc registered [ 1395.847823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1434.698175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1434.726102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1435.971907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1435.996353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1436.062201] Lustre: lustre-MDT0000: new disk, initializing [ 1436.144283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.178581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1441.197595] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1452.553926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.686876] Lustre: 46022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1452.734654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1452.738677] Lustre: Skipped 1 previous similar message [ 1452.803679] Lustre: lustre-MDT0001: new disk, initializing [ 1452.893132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1452.930811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1452.951678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1457.789447] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1468.330801] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1475.006591] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1481.773295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1482.035862] Lustre: lustre-OST0000: new disk, initializing [ 1482.039607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1482.047366] Lustre: 47990:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1482.177413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.974705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1488.019175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1488.170405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1489.356081] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1499.972955] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1504.869949] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1505.048800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1510.145357] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1510.382542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1513.958193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1513.958540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.960842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.978695] LustreError: Skipped 1 previous similar message [ 1514.019120] Lustre: Skipped 1 previous similar message [ 1519.083612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1519.090699] Lustre: Skipped 1 previous similar message [ 1524.201845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1524.214795] Lustre: Skipped 1 previous similar message [ 1527.263183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1527.432694] Lustre: server umount lustre-OST0000 complete [ 1534.435388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.438713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1534.461501] Lustre: Skipped 2 previous similar messages [ 1539.553074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.566245] Lustre: Skipped 2 previous similar messages [ 1546.207144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1546.533170] Lustre: server umount lustre-MDT0000 complete [ 1550.300286] LustreError: 47989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569731 with bad export cookie 6760902794280220135 [ 1550.312690] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1550.801183] Lustre: server umount lustre-MDT0001 complete [ 1560.620636] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:55:38 (1788569738) [ 1567.611024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.022637] LustreError: 50454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.037702] LustreError: 50454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1568.082641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.346594] LustreError: 50455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1573.751967] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1577.441919] LustreError: 50454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.254712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1582.566657] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1582.590885] Lustre: Skipped 2 previous similar messages [ 1582.729328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1587.408046] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1595.410447] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1601.432506] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1608.253820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1608.702477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1615.983552] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1624.043521] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1631.945728] LustreError: 52984:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1632.880781] Lustre: Modifying parameter general.timeout=1 in log params [ 1634.710491] Lustre: Modifying parameter general.timeout=3 in log params [ 1634.718796] Lustre: Skipped 1 previous similar message [ 1636.732166] Lustre: Modifying parameter general.timeout=5 in log params [ 1636.740987] Lustre: Skipped 1 previous similar message [ 1641.326530] Lustre: Modifying parameter general.timeout=10 in log params [ 1641.332988] Lustre: Skipped 4 previous similar messages [ 1649.390221] Lustre: Modifying parameter general.timeout=19 in log params [ 1649.394631] Lustre: Skipped 8 previous similar messages [ 1666.198706] Lustre: Modifying parameter general.timeout=39 in log params [ 1666.202518] Lustre: Skipped 19 previous similar messages [ 1698.799988] Lustre: Modifying parameter general.timeout=75 in log params [ 1698.802506] Lustre: Skipped 35 previous similar messages [ 1758.941530] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:58:56 (1788569936) [ 1761.702813] Lustre: 59925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1763.573621] Lustre: 60023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1764.485595] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1764.491791] Lustre: Skipped 58 previous similar messages [ 1766.925506] Lustre: 60221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1766.937543] Lustre: 60221:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1796.839273] Lustre: 61796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1796.845577] Lustre: 61796:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1796.867491] LustreError: 61796:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1800.921373] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:59:39 (1788569979) [ 1817.057922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.080126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1817.082409] Lustre: Skipped 1 previous similar message [ 1820.100332] Lustre: server umount lustre-MDT0000 complete [ 1822.183037] LustreError: 52945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.200434] LustreError: 52945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1823.858360] LustreError: 50435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570004 with bad export cookie 6760902794280221017 [ 1823.867442] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1823.870948] LustreError: 50435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1824.385872] Lustre: server umount lustre-MDT0001 complete [ 1838.915730] Lustre: server umount lustre-OST0000 complete [ 1866.494532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.796663] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1867.042520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.948894] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1879.942883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.091130] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1880.116721] Lustre: 64326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1886.226791] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1893.127975] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1898.982693] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1905.797105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1906.104375] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1906.397887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.409612] Lustre: Skipped 1 previous similar message [ 1912.820529] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1919.761273] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1922.161304] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1922.172293] Lustre: Skipped 19 previous similar messages [ 1922.292577] Lustre: 66908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1933.797268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1933.803914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.822637] Lustre: Skipped 1 previous similar message [ 1933.825982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1933.841369] Lustre: Skipped 2 previous similar messages [ 1936.343454] Lustre: server umount lustre-OST0000 complete [ 1941.984883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1941.990867] LustreError: Skipped 1 previous similar message [ 1941.993743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.007815] Lustre: Skipped 1 previous similar message [ 1955.296620] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1955.619109] Lustre: server umount lustre-MDT0000 complete [ 1959.508431] LustreError: 64321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570140 with bad export cookie 6760902794280223978 [ 1959.512462] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1959.518270] LustreError: 64321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1959.971754] Lustre: server umount lustre-MDT0001 complete [ 1973.097931] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1975.890950] Key type lgssc unregistered [ 1976.263759] LNet: 68118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.278026] LNetError: 68118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.323711] LNet: Removed LNI 192.168.202.152@tcp [ 1978.508349] Key type .llcrypt unregistered [ 1978.511240] Key type ._llcrypt unregistered [ 2006.067474] Key type ._llcrypt registered [ 2006.069606] Key type .llcrypt registered [ 2006.235618] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2007.284806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2007.392385] alg: No test for adler32 (adler32-zlib) [ 2008.435244] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2008.673877] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2010.359206] Key type lgssc registered [ 2011.553178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2019.552290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2019.572460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.165169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.568915] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2030.568607] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.595916] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2035.680774] LustreError: 69527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.157410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.959638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2042.066981] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2050.549195] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2057.168937] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2063.705060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2064.468389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.851945] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2078.868639] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2091.506290] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2102.762145] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.768684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.791076] Lustre: Skipped 2 previous similar messages [ 2102.813693] Lustre: Skipped 2 previous similar messages [ 2107.878329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2107.882058] Lustre: Skipped 2 previous similar messages [ 2108.718292] Lustre: server umount lustre-MDT0000 complete [ 2112.630255] LustreError: 69510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570293 with bad export cookie 5064935297223377381 [ 2112.635229] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2112.644851] LustreError: 69510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2113.079234] Lustre: server umount lustre-MDT0001 complete [ 2128.568842] Lustre: server umount lustre-OST0000 complete [ 2155.275162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.508065] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2155.533485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2155.890738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.777430] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2168.820449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.900288] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2168.944966] Lustre: 74399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2169.349322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2174.030485] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2181.282799] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2186.889337] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2193.901738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2194.194528] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2194.744974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.089145] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2209.353950] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2211.133401] Lustre: 76973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2213.043371] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2223.088077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2223.098160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.129405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2223.598337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2225.723550] Lustre: server umount lustre-OST0000 complete [ 2230.752664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2230.761159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.781350] Lustre: Skipped 1 previous similar message [ 2230.786224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2230.791660] Lustre: Skipped 1 previous similar message [ 2233.826968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.473176] Lustre: server umount lustre-MDT0000 complete [ 2238.944807] LustreError: 74706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.965240] LustreError: 74706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2240.769289] LustreError: 77233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570421 with bad export cookie 5064935297223378221 [ 2240.774700] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.781516] LustreError: 77233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2241.179342] Lustre: server umount lustre-MDT0001 complete [ 2255.089838] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2258.760740] Key type lgssc unregistered [ 2259.027685] LNet: 78188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2259.033362] LNetError: 78188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2259.053439] LNet: Removed LNI 192.168.202.152@tcp [ 2260.109248] Key type .llcrypt unregistered [ 2260.111356] Key type ._llcrypt unregistered [ 2285.575581] Key type ._llcrypt registered [ 2285.577915] Key type .llcrypt registered [ 2285.703874] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2287.141483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2287.158962] alg: No test for adler32 (adler32-zlib) [ 2288.318217] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2288.512614] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2290.225744] Key type lgssc registered [ 2291.169192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2298.083914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2298.095510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2299.557832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2304.657800] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2310.125456] LustreError: 79602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.155962] LustreError: 79602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2313.701098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.542489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2319.390445] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2328.307181] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2335.171761] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2342.953269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2343.785740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.479325] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2360.575892] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2367.774281] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2369.155276] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2370.162775] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2372.235667] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2372.237776] Lustre: Skipped 1 previous similar message [ 2376.376279] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2376.380571] Lustre: Skipped 3 previous similar messages [ 2385.131171] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2385.141304] Lustre: Skipped 8 previous similar messages [ 2401.382152] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2401.389486] Lustre: Skipped 15 previous similar messages [ 2433.579729] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2433.583880] Lustre: Skipped 32 previous similar messages [ 2474.759879] Lustre: 87707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2477.492884] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:10:55 (1788570655) [ 2479.737725] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2481.926516] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:10:59 (1788570659) [ 2487.270643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2487.274257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.309307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2487.319766] Lustre: Skipped 1 previous similar message [ 2492.358331] Lustre: server umount lustre-OST0000 complete [ 2497.505775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.519105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2497.531732] Lustre: Skipped 2 previous similar messages [ 2497.542583] Lustre: Skipped 1 previous similar message [ 2502.626900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2502.643325] Lustre: Skipped 1 previous similar message [ 2503.077566] Lustre: server umount lustre-MDT0000 complete [ 2507.745000] LustreError: 80332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.781055] LustreError: 80332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2508.245629] LustreError: 81455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570688 with bad export cookie 10365702185555122651 [ 2508.252436] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2508.270913] LustreError: 81455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2508.764303] Lustre: server umount lustre-MDT0001 complete [ 2522.641639] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2525.848439] Key type lgssc unregistered [ 2526.083549] LNet: 88956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2526.100640] LNetError: 88956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.137527] LNet: Removed LNI 192.168.202.152@tcp [ 2526.992846] Key type .llcrypt unregistered [ 2526.995571] Key type ._llcrypt unregistered [ 2531.356245] Key type ._llcrypt registered [ 2531.357365] Key type .llcrypt registered [ 2531.442850] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2536.294097] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules [ 2537.577577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2537.773031] alg: No test for adler32 (adler32-zlib) [ 2538.816960] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2539.125775] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2540.935415] Key type lgssc registered [ 2542.686428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2544.335700] LustreError: 90191:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2564.153696] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2567.518683] LustreError: 90191:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2575.038286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2575.061796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.717089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.669072] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2587.104573] LustreError: 91202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.128407] LustreError: 91202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2592.229116] LustreError: 91203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.150553] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:12:50 (1788570770) [ 2597.345490] LustreError: 91202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.224307] LustreError: 91234:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2599.244073] LustreError: 91234:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2599.253922] LustreError: 91234:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2599.591844] Lustre: server umount lustre-MDT0000 complete [ 2615.887833] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2619.208509] Key type lgssc unregistered [ 2619.661740] LNet: 92729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2619.667374] LNetError: 92729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2619.698899] LNet: Removed LNI 192.168.202.152@tcp [ 2620.864902] Key type .llcrypt unregistered [ 2620.867122] Key type ._llcrypt unregistered [ 2641.454676] Key type ._llcrypt registered [ 2641.456921] Key type .llcrypt registered [ 2641.542271] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2642.535869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2642.612341] alg: No test for adler32 (adler32-zlib) [ 2643.703586] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2643.934569] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2645.591793] Key type lgssc registered [ 2646.932263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2657.494723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2657.527101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.984470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.323200] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2668.525686] LustreError: 93944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.546666] LustreError: 93944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2673.632418] LustreError: 93945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.682975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.755347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2681.101986] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2690.401378] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2697.710543] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2706.285770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2707.447755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.254602] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2725.500049] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2788.474396] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 21:16:05 (1788570965) [ 2791.130784] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2793.934853] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 21:16:10 (1788570970) [ 2800.612121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.627901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.636564] Lustre: Skipped 2 previous similar messages [ 2800.656982] Lustre: Skipped 2 previous similar messages [ 2805.732534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2805.751033] Lustre: Skipped 2 previous similar messages [ 2806.441142] Lustre: server umount lustre-MDT0000 complete [ 2810.853913] LustreError: 96739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.870107] LustreError: 96739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2811.643556] LustreError: 93926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570992 with bad export cookie 14136857081815749976 [ 2811.647887] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.659051] LustreError: 93926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2812.626402] Lustre: server umount lustre-MDT0001 complete [ 2828.414416] Lustre: server umount lustre-OST0000 complete [ 2842.729201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.312146] LustreError: 98244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.601954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.749870] LustreError: 98245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.569940] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2858.977112] LustreError: 98245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2859.005750] LustreError: 98245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2860.094959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.797061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2866.633676] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2875.721875] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2882.127441] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2897.977523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.256606] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2898.261211] LustreError: 98234:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2898.292046] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2898.299469] LustreError: 100334:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2898.306498] LustreError: 100334:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2898.311191] LustreError: 100334:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2898.372988] Lustre: server umount lustre-OST0000 complete [ 2898.377690] LustreError: 100334:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2908.831748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2909.082986] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2909.088945] LustreError: 98233:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2909.108698] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2909.129557] LustreError: 100802:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2909.138074] LustreError: 100802:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2909.144246] LustreError: 100802:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2909.212599] Lustre: server umount lustre-OST0000 complete [ 2909.214792] LustreError: 100802:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2920.848956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.039625] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2921.049454] Lustre: Found index 0 for lustre-OST0000, updating log [ 2921.064859] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2921.183233] Lustre: lustre-OST0000: new disk, initializing [ 2921.186910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2921.193560] Lustre: 101336:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2921.662684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.009730] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2934.995977] Lustre: Failing over lustre-OST0000 [ 2935.172513] Lustre: server umount lustre-OST0000 complete [ 2942.434036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2942.445733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.473304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2944.482057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2944.482603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.192055] Lustre: server umount lustre-MDT0000 complete [ 2949.605401] LustreError: 99988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.643043] LustreError: 99988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2950.910422] LustreError: 98225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571131 with bad export cookie 14136857081815752713 [ 2950.916792] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2950.922702] LustreError: 98225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2960.681669] Lustre: server umount lustre-MDT0001 complete [ 2971.657433] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:19:09 (1788571149) [ 2990.681993] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3001.609081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3001.982415] LustreError: 103687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.083107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.628294] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3016.900232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.614262] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3027.062872] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3034.885529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3035.603561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3035.609052] Lustre: Skipped 1 previous similar message [ 3035.616494] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3042.807353] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3042.861618] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3042.862206] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3043.423676] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3052.875668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3052.980627] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3053.065450] Lustre: lustre-OST0001: new disk, initializing [ 3053.072392] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3053.079798] Lustre: 106005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3061.100664] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3063.512811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3063.525850] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3063.660941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3069.837383] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3081.402401] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3089.397716] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.410532] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3089.423423] Lustre: Skipped 1 previous similar message [ 3089.429674] Lustre: Skipped 1 previous similar message [ 3092.320609] Lustre: server umount lustre-MDT0001 complete [ 3094.526738] LustreError: 103688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.546408] LustreError: 103688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3105.176708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.493504] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3105.524942] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3105.654229] Lustre: 103676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3106.944101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3106.956277] Lustre: Skipped 1 previous similar message [ 3107.696505] LustreError: 107543:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3107.701727] LustreError: 107543:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3107.715906] LustreError: 103679:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3107.850695] LustreError: 107552:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.152@tcp: cfg command failed: rc = -17 [ 3107.878180] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3107.878180] [ 3112.466790] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3113.969096] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3125.467157] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 21:21:42 (1788571302) [ 3146.654083] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3168.431352] Lustre: 109620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3193.365328] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3197.447515] Lustre: 110756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3285.491514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3285.503267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3285.524231] Lustre: Skipped 1 previous similar message [ 3285.541605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3285.556164] Lustre: Skipped 2 previous similar messages [ 3288.893559] Lustre: server umount lustre-MDT0000 complete [ 3291.626807] LustreError: 105503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.643769] LustreError: 105503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3293.227297] LustreError: 106004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571473 with bad export cookie 14136857081815753714 [ 3293.250545] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3293.255126] LustreError: 106004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3293.905514] Lustre: server umount lustre-MDT0001 complete [ 3309.345671] Lustre: server umount lustre-OST0000 complete [ 3312.996877] Lustre: 93255:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788571477/real 1788571477] req@ffff97262dc1ca80 x1875452440732032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788571493 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3313.034084] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.052028] Lustre: Skipped 3 previous similar messages [ 3313.922198] Lustre: server umount lustre-OST0001 complete [ 3321.847310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3335.075344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3345.994773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3357.973744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3382.635438] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3396.813726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.866918] Lustre: lustre-MDT0000: reset Object Index mappings [ 3397.650038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.706767] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3413.428188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.458142] Lustre: lustre-MDT0001: reset Object Index mappings [ 3421.560350] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3424.934852] Lustre: 115856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3432.211286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3438.414446] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3441.156880] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3443.523480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3452.166566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3458.076278] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3458.078046] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3460.954686] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3468.582755] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3473.169927] Lustre: 117735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3531.247778] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 21:28:28 (1788571708) [ 3557.351266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3557.360550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3557.384408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3557.395141] Lustre: Skipped 4 previous similar messages [ 3562.887300] Lustre: server umount lustre-MDT0000 complete [ 3565.556111] LustreError: 117761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3565.592139] LustreError: 117761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3568.333624] LustreError: 114688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571749 with bad export cookie 14136857081815800572 [ 3568.350655] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3568.374108] LustreError: 114688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3575.145254] Lustre: server umount lustre-MDT0001 complete [ 3594.207131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3594.541774] Lustre: server umount lustre-OST0000 complete [ 3604.823876] Lustre: server umount lustre-OST0001 complete [ 3615.729776] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3624.920955] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3666.566225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.906086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3666.946914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3667.056827] Lustre: lustre-MDT0000: new disk, initializing [ 3667.145445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3667.147841] Lustre: Skipped 3 previous similar messages [ 3667.177603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3671.880970] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3679.216302] Lustre: server umount lustre-MDT0000 complete [ 3690.245248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3690.621620] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3690.641578] Lustre: 122009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3690.669778] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3696.854446] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3702.940174] Lustre: server umount lustre-MDT0000 complete [ 3714.251424] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 21:31:31 (1788571891) [ 3715.998893] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3717.864797] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 21:31:35 (1788571895) [ 3738.010593] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3750.531923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.496863] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3767.512943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.662436] Lustre: 124062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3767.738236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3767.745433] Lustre: Skipped 1 previous similar message [ 3767.821374] Lustre: lustre-MDT0001: new disk, initializing [ 3767.951616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3767.971946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3773.352899] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3779.308987] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3787.265063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3787.600297] Lustre: lustre-OST0000: new disk, initializing [ 3787.603914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3787.612469] Lustre: 125640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3789.399423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3789.423489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3789.605114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3795.916875] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3807.900710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3808.096934] Lustre: lustre-OST0001: new disk, initializing [ 3808.099311] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3808.105317] Lustre: 126511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3809.461617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3809.481523] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3809.580735] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3814.659951] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3824.940582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3828.204324] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4233.449250] Lustre: 132934:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4233.457874] Lustre: 132934:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4233.463890] Lustre: 132934:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4233.467801] Lustre: 132934:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4233.471695] Lustre: 132934:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4233.477757] Lustre: 132934:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4438.561271] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:43:37 (1788572617) [ 4454.367993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4454.371065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.375390] Lustre: Skipped 5 previous similar messages [ 4454.377105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4454.379195] Lustre: Skipped 8 previous similar messages [ 4456.184929] Lustre: server umount lustre-MDT0000 complete [ 4457.443188] LustreError: 125639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572638 with bad export cookie 14136857081815869879 [ 4457.450301] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4459.999983] LustreError: 127309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4460.000627] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.004631] LustreError: 127309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4460.010595] Lustre: Skipped 3 previous similar messages [ 4460.012957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4460.014824] Lustre: Skipped 4 previous similar messages [ 4463.684772] Lustre: server umount lustre-MDT0001 complete [ 4471.046903] Lustre: server umount lustre-OST0000 complete [ 4478.483241] Lustre: server umount lustre-OST0001 complete [ 4480.610723] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4483.460748] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4495.385790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.471299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4495.480648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4495.517114] Lustre: lustre-MDT0000: new disk, initializing [ 4495.537840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.539949] Lustre: Skipped 5 previous similar messages [ 4495.544883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4496.866688] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4501.137152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4501.182821] Lustre: 135967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4501.212237] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4501.215307] Lustre: Skipped 1 previous similar message [ 4501.252460] Lustre: lustre-MDT0001: new disk, initializing [ 4501.309626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4501.313968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4502.740718] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4506.604850] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4508.602607] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4510.999832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4511.125453] Lustre: lustre-OST0000: new disk, initializing [ 4511.127360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4511.130427] Lustre: 137936:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4512.569387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4512.572669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4512.616198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4513.282405] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4517.071969] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4520.339505] Lustre: lustre-MDD0000: changelog on [ 4521.270674] Lustre: lustre-MDD0001: changelog on [ 4847.581896] Lustre: 139698:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4875.544365] Lustre: lustre-MDD0001: changelog off [ 4876.833833] Lustre: lustre-MDD0000: changelog off [ 4881.378384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.379716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4881.380185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4881.380190] Lustre: Skipped 2 previous similar messages [ 4881.382137] Lustre: Skipped 2 previous similar messages [ 4881.384314] LustreError: Skipped 1 previous similar message [ 4884.507045] Lustre: server umount lustre-OST0000 complete [ 4900.319126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4900.436546] Lustre: server umount lustre-MDT0000 complete [ 4901.741426] LustreError: 135962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573082 with bad export cookie 14136857081816644996 [ 4901.744681] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4901.745189] LustreError: 135962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4901.873801] Lustre: server umount lustre-MDT0001 complete [ 4905.537411] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4906.610328] Key type lgssc unregistered [ 4906.737365] LNet: 146676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4906.741300] LNetError: 146676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4906.751444] LNet: Removed LNI 192.168.202.152@tcp [ 4907.087111] Key type .llcrypt unregistered [ 4907.088063] Key type ._llcrypt unregistered [ 4913.492698] Key type ._llcrypt registered [ 4913.494352] Key type .llcrypt registered [ 4913.536100] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4919.537207] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4920.052816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4920.059604] alg: No test for adler32 (adler32-zlib) [ 4920.937760] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4921.034777] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4922.623183] Key type lgssc registered [ 4923.055896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4934.702372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4934.708965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.809908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4935.821477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4935.866211] Lustre: lustre-MDT0000: new disk, initializing [ 4935.887140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4935.894360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4937.314477] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4941.489039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.530400] Lustre: 149843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4941.555528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4941.557599] Lustre: Skipped 1 previous similar message [ 4941.595363] Lustre: lustre-MDT0001: new disk, initializing [ 4941.625906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4941.637697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4941.640343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4942.969770] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4946.664122] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4948.511611] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4950.837067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4950.939428] Lustre: lustre-OST0000: new disk, initializing [ 4950.941250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4950.944372] Lustre: 151810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4950.977727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.055909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4952.059130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4952.094756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4953.023815] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4956.799819] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4959.059053] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_module obdecho/obdecho [ 4959.651414] LustreError: 152814:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4959.654355] LustreError: 152814:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4960.247524] LustreError: 152910:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 4967.393215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4967.394199] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4967.396714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4967.399570] Lustre: Skipped 2 previous similar messages [ 4967.401155] Lustre: Skipped 1 previous similar message [ 4969.636252] Lustre: server umount lustre-MDT0000 complete [ 4970.838216] LustreError: 149837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573151 with bad export cookie 13331251969903904720 [ 4970.841934] LustreError: 149837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4970.842042] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4970.941863] Lustre: server umount lustre-MDT0001 complete [ 4981.406897] Lustre: server umount lustre-OST0000 complete [ 4988.358750] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4991.554103] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5003.097875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.192168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5003.203222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5003.240963] Lustre: lustre-MDT0000: new disk, initializing [ 5003.261645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5003.267294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5004.580743] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5008.758474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.800229] Lustre: 156617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5008.825074] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5008.826854] Lustre: Skipped 1 previous similar message [ 5008.862216] Lustre: lustre-MDT0001: new disk, initializing [ 5008.894455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5008.908942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5008.912253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5010.294307] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5014.015923] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5015.872378] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5018.184359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5018.288450] Lustre: lustre-OST0000: new disk, initializing [ 5018.291239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5018.294150] Lustre: 158583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5018.322979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5020.260668] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5024.003134] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5025.490065] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5025.786741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5025.791983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5025.824031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5026.577904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5028.026779] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5028.118279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5030.883894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5030.884021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.885962] LustreError: Skipped 1 previous similar message [ 5030.886533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5030.889283] Lustre: Skipped 1 previous similar message [ 5030.893223] Lustre: Skipped 2 previous similar messages [ 5034.835213] Lustre: server umount lustre-OST0000 complete [ 5039.583969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5039.586470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5039.590991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5042.273091] Lustre: server umount lustre-MDT0000 complete [ 5043.534296] LustreError: 158582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573224 with bad export cookie 13331251969903905665 [ 5043.537384] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5043.538979] LustreError: 158582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5043.657580] Lustre: server umount lustre-MDT0001 complete [ 5044.886490] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:53:43 (1788573223) [ 5045.421604] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5046.019061] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:53:44 (1788573224) [ 5048.620691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.806051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.227116] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5053.045918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.536339] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5057.004951] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5058.850296] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5061.054705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5063.184703] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5065.742258] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5079.010305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5079.010521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5079.011856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5079.011862] Lustre: Skipped 2 previous similar messages [ 5079.013800] Lustre: Skipped 3 previous similar messages [ 5084.128621] Lustre: server umount lustre-MDT0000 complete [ 5084.129430] LustreError: 161738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5084.135937] LustreError: 161738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5085.379287] LustreError: 161024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573266 with bad export cookie 13331251969903906533 [ 5085.384031] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5085.384748] LustreError: 161024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5086.394676] LustreError: 160994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5086.400785] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [ 5086.402704] Lustre: Skipped 3 previous similar messages [ 5089.248214] LustreError: 160993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5089.248425] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.514623] LustreError: 160995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5091.520112] LustreError: 160995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5091.573134] Lustre: server umount lustre-MDT0001 complete [ 5093.813727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5097.955335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5098.041458] LustreError: 164860:0:(llog.c:1655:llog_backup()) MGC192.168.202.152@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5098.043995] Lustre: 164860:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.152@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5115.872114] LustreError: 164865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5126.111878] LustreError: 164887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5126.116925] LustreError: 164887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5142.714545] LustreError: 164889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5142.720482] LustreError: 164889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5159.007337] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5159.010214] Lustre: 164860:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.152@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5175.776025] LustreError: 164867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.781037] LustreError: 164867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [ 5199.854939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5199.857688] Lustre: Skipped 2 previous similar messages [ 5201.285200] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5204.188685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.964251] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5205.983581] LustreError: 148070:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c6de7505 x1875454828052608/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5207.408212] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5209.835355] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5210.336653] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5211.108609] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5212.183770] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5213.714267] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 5220.571522] Lustre: server umount lustre-MDT0000 complete [ 5221.344600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.347185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5221.348598] Lustre: Skipped 1 previous similar message [ 5221.845211] LustreError: 163803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573402 with bad export cookie 13331251969903912973 [ 5221.848260] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5221.849916] LustreError: 163803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5221.956133] Lustre: server umount lustre-MDT0001 complete [ 5233.560186] Lustre: server umount lustre-OST0000 complete [ 5240.430564] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5243.768201] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5255.704684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.798677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5255.809137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5255.846059] Lustre: lustre-MDT0000: new disk, initializing [ 5255.872587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5257.264738] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5261.389935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.429363] Lustre: 169935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5261.531978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5262.895752] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5266.574533] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5268.454409] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5270.724635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5270.836289] Lustre: 171901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5270.867855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5270.869790] Lustre: Skipped 3 previous similar messages [ 5272.449887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5272.498357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5272.867301] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5276.661615] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5278.178146] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5278.251979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5279.740292] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5279.820493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5282.575143] Lustre: server umount lustre-OST0000 complete [ 5287.392105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5287.394563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.398513] Lustre: Skipped 1 previous similar message [ 5287.400347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5287.402647] Lustre: Skipped 3 previous similar messages [ 5298.143151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5298.210040] Lustre: server umount lustre-MDT0000 complete [ 5299.167129] Lustre: 148071:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788573463/real 1788573463] req@000000002dd43525 x1875454828097536/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788573479 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5299.176670] LustreError: 169945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5299.184311] LustreError: 169945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 5299.545178] LustreError: 169972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573480 with bad export cookie 13331251969903929773 [ 5299.548142] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5299.549757] LustreError: 169972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5299.683660] Lustre: server umount lustre-MDT0001 complete [ 5302.777878] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:58:01 (1788573481) [ 5305.103339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5306.706086] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5309.621936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.162720] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5313.562202] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5315.385699] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5317.633604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5319.767957] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5322.147634] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5336.437765] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:58:35 (1788573515) [ 5340.641642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5340.644283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.648297] Lustre: Skipped 2 previous similar messages [ 5340.650465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5340.652318] Lustre: Skipped 5 previous similar messages [ 5343.506455] Lustre: server umount lustre-OST0000 complete [ 5359.071066] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5359.134948] Lustre: server umount lustre-MDT0000 complete [ 5360.392255] LustreError: 174291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573541 with bad export cookie 13331251969903930641 [ 5360.395770] LustreError: 174291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5360.396204] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5360.517726] Lustre: server umount lustre-MDT0001 complete [ 5364.134925] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5365.222506] Key type lgssc unregistered [ 5365.358406] LNet: 178057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5365.360761] LNetError: 178057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5365.368491] LNet: Removed LNI 192.168.202.152@tcp [ 5365.727777] Key type .llcrypt unregistered [ 5365.729484] Key type ._llcrypt unregistered [ 5373.385508] Key type ._llcrypt registered [ 5373.386604] Key type .llcrypt registered [ 5373.426592] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5373.768559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5373.784332] alg: No test for adler32 (adler32-zlib) [ 5374.644565] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 5374.738071] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5376.327206] Key type lgssc registered [ 5376.706781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5379.346034] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5379.353017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5394.784793] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5399.903497] LustreError: 179272:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 4: rc = -110 [ 5405.024737] LustreError: 179272:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5405.029894] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5405.032471] LustreError: MGC192.168.202.152@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5405.037024] LustreError: 179272:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5405.040607] LustreError: 179272:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 5405.043723] LustreError: 179272:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5405.067417] Lustre: server umount lustre-OST0000 complete [ 5405.068859] LustreError: 179272:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5407.910962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.055070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5410.439085] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5413.363541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5413.490295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5414.985455] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5417.464900] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5419.407959] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5421.674014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5421.761918] LustreError: 181584:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5421.765316] Lustre: 181584:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.152@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5421.806597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.779708] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5426.255194] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5428.703986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5428.708344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.713873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5429.216515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5429.218578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5429.222588] Lustre: Skipped 2 previous similar messages [ 5433.116257] Lustre: server umount lustre-MDT0000 complete [ 5434.338179] LustreError: 181965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5434.343138] LustreError: 181965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5434.366057] LustreError: 179753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573615 with bad export cookie 13402135031552545003 [ 5434.369421] LustreError: 179753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5434.369507] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5434.488068] Lustre: server umount lustre-MDT0001 complete [ 5446.042636] Lustre: server umount lustre-OST0000 complete [ 5449.130651] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5452.408157] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5464.551714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.642523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5464.654817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5464.694882] Lustre: lustre-MDT0000: new disk, initializing [ 5464.717817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.724591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5466.136220] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5470.286277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.329297] Lustre: 185100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5470.352868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5470.354760] Lustre: Skipped 1 previous similar message [ 5470.394252] Lustre: lustre-MDT0001: new disk, initializing [ 5470.425241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5470.436262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5470.439248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5471.904721] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5475.619106] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5477.453511] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5479.721226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5479.840846] Lustre: lustre-OST0000: new disk, initializing [ 5479.842832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5479.845505] Lustre: 187069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5479.877955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5481.944571] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5483.129355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5483.132874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5483.166836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5485.639212] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5487.041247] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5487.118897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5488.547982] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5488.633139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5493.728820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5493.731381] LustreError: Skipped 1 previous similar message [ 5493.731402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.733173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5493.736185] Lustre: Skipped 2 previous similar messages [ 5493.741813] Lustre: Skipped 1 previous similar message [ 5495.381675] Lustre: server umount lustre-OST0000 complete [ 5498.848955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5498.849613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5498.853771] Lustre: Skipped 1 previous similar message [ 5502.756098] Lustre: server umount lustre-MDT0000 complete [ 5503.968109] LustreError: 186380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.973937] LustreError: 186380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5504.032226] LustreError: 185092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573684 with bad export cookie 13402135031552545703 [ 5504.035015] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5504.037427] LustreError: 185092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5504.157451] Lustre: server umount lustre-MDT0001 complete [ 5507.163761] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 22:01:26 (1788573686) [ 5513.302938] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5514.417635] Key type lgssc unregistered [ 5514.553784] LNet: 190018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5514.556994] LNetError: 190018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5514.564364] LNet: Removed LNI 192.168.202.152@tcp [ 5514.910768] Key type .llcrypt unregistered [ 5514.911948] Key type ._llcrypt unregistered [ 5522.264835] Key type ._llcrypt registered [ 5522.265929] Key type .llcrypt registered [ 5522.305131] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5522.766441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5522.788870] alg: No test for adler32 (adler32-zlib) [ 5523.656387] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 5523.749533] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5525.327200] Key type lgssc registered [ 5525.677874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5528.314821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5528.321178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5529.463252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5530.808296] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5533.627273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.755626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5535.080147] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5537.456054] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5539.188664] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5541.598924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5541.664961] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5541.666300] LustreError: 193070:0:(llog.c:1696:llog_backup()) MGC192.168.202.152@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5541.668603] Lustre: 193070:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.152@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5541.702702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.532354] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5545.942410] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5546.688420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5546.690948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5546.692021] Lustre: Skipped 2 previous similar messages [ 5546.694395] Lustre: Skipped 1 previous similar message [ 5552.096672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5552.098966] Lustre: Skipped 3 previous similar messages [ 5552.859967] Lustre: server umount lustre-MDT0000 complete [ 5554.031151] LustreError: 191190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573734 with bad export cookie 9408785112237097235 [ 5554.035528] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5554.036799] LustreError: 191190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5554.136573] Lustre: server umount lustre-MDT0001 complete [ 5565.668119] Lustre: server umount lustre-OST0000 complete [ 5568.623954] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5571.820244] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5583.657729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.745615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5583.755525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5583.792140] Lustre: lustre-MDT0000: new disk, initializing [ 5583.815888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5583.821955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5585.261362] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5589.496843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5589.549164] Lustre: 196590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5589.572707] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5589.573833] Lustre: Skipped 1 previous similar message [ 5589.574965] LustreError: 197372:0:(llog.c:1696:llog_backup()) MGC192.168.202.152@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5589.577276] LustreError: 197372:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5589.579134] Lustre: 197372:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.152@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5589.586669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5589.588231] Lustre: Skipped 1 previous similar message [ 5589.626923] Lustre: lustre-MDT0001: new disk, initializing [ 5589.666312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5589.679464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5589.683196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5591.092161] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5594.709249] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5596.472361] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5598.744676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5598.854786] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5598.856109] Lustre: Skipped 1 previous similar message [ 5598.857406] LustreError: 198534:0:(llog.c:1696:llog_backup()) MGC192.168.202.152@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5598.860175] LustreError: 198534:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5598.862298] Lustre: 198534:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.152@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5598.865201] Lustre: 198534:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5598.876346] Lustre: lustre-OST0000: new disk, initializing [ 5598.877969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5598.881716] Lustre: 198557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5598.920751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5600.916108] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5604.643968] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5605.887046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5605.890120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5605.924996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5606.126667] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5606.196315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5607.596801] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5607.678565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5610.977096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5610.979029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.979703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5610.983796] Lustre: Skipped 1 previous similar message [ 5614.426148] Lustre: server umount lustre-OST0000 complete [ 5616.096830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.097581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5616.100931] Lustre: Skipped 1 previous similar message [ 5616.102616] Lustre: Skipped 1 previous similar message [ 5621.216046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5621.217876] Lustre: Skipped 2 previous similar messages [ 5621.846418] Lustre: server umount lustre-MDT0000 complete [ 5623.097080] LustreError: 197387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573803 with bad export cookie 9408785112237097935 [ 5623.100086] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5623.102258] LustreError: 197387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5623.234550] Lustre: server umount lustre-MDT0001 complete [ 5626.111667] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 22:03:25 (1788573805) [ 5631.648203] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5635.392960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5635.535098] LustreError: 201678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.544045] LustreError: 201678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5635.547818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.890848] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5639.676776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.230327] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5642.221571] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5644.446379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5646.579836] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5649.496845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5649.568300] Lustre: lustre-OST0001: new disk, initializing [ 5649.570467] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5649.574909] Lustre: 203983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5651.617785] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5654.530256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5654.534437] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5654.565062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5657.128793] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5661.835634] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5662.474211] Lustre: DEBUG MARKER: ADD OST3 [ 5664.182897] loop: module loaded [ 5664.202212] loop0: detected capacity change from 0 to 204800000 [ 5664.293794] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5664.297142] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5664.302481] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5666.360382] loop0: detected capacity change from 0 to 204800000 [ 5668.270254] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5668.343269] Lustre: lustre-OST0003: new disk, initializing [ 5668.344975] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5668.348918] Lustre: 205714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5668.402035] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5668.404149] Lustre: Skipped 3 previous similar messages [ 5669.319610] Lustre: DEBUG MARKER: STOP OST3 [ 5669.975634] Lustre: Failing over lustre-OST0003 [ 5670.024798] Lustre: server umount lustre-OST0003 complete [ 5670.217468] LustreError: 203343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5670.659672] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5674.554797] Lustre: DEBUG MARKER: START OST3 again [ 5675.498774] LustreError: 203163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.503458] LustreError: 203163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5676.877672] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5678.940587] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5682.165844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5682.170649] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5682.180034] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5692.580077] Lustre: Failing over lustre-OST0003 [ 5692.636571] Lustre: server umount lustre-OST0003 complete [ 5697.511997] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5697.512783] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.514689] LustreError: Skipped 2 previous similar messages [ 5697.517721] LustreError: 203162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5697.517727] LustreError: 203162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5697.521034] Lustre: Skipped 1 previous similar message [ 5702.627340] LustreError: 203162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.631899] LustreError: 203162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5706.720085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5706.723182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.727688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5711.390028] Lustre: server umount lustre-MDT0000 complete [ 5712.632339] LustreError: 201660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573893 with bad export cookie 9408785112237098803 [ 5712.634135] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5712.635734] LustreError: 201660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5712.864526] LustreError: 201673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5712.869537] LustreError: 201673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5718.901812] Lustre: server umount lustre-MDT0001 complete [ 5720.217271] Lustre: server umount lustre-OST0000 complete [ 5721.509365] Lustre: server umount lustre-OST0001 complete [ 5723.715439] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5726.741809] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5739.467371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.557841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5739.570162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5739.607129] Lustre: lustre-MDT0000: new disk, initializing [ 5739.628149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.631454] Lustre: Skipped 1 previous similar message [ 5739.637573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5741.054442] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5745.238187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.284107] Lustre: 211124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5745.404104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5746.864697] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5750.658754] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5752.523206] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5754.807470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5754.936604] Lustre: 213093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5756.476414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5756.478850] Lustre: Skipped 1 previous similar message [ 5756.480383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5756.520186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5757.018912] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5760.749622] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5762.262849] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5762.346337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5763.807416] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5763.890842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5766.627411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5766.628589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5766.629094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5766.629103] Lustre: Skipped 8 previous similar messages [ 5766.637106] Lustre: Skipped 6 previous similar messages [ 5770.639435] Lustre: server umount lustre-OST0000 complete [ 5776.351933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5778.078374] Lustre: server umount lustre-MDT0000 complete [ 5779.333710] LustreError: 211119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573960 with bad export cookie 9408785112237101036 [ 5779.338891] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5779.467056] Lustre: server umount lustre-MDT0001 complete [ 5780.655702] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 22:05:59 (1788573959) [ 5785.886557] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5789.372784] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5801.431627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.524909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5801.537412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5801.540276] Lustre: Skipped 3 previous similar messages [ 5801.585099] Lustre: lustre-MDT0000: new disk, initializing [ 5801.587178] Lustre: Skipped 2 previous similar messages [ 5801.616253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5803.061444] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5807.293186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5807.333379] Lustre: 217690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5807.434744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5808.813232] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5812.614783] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5814.534101] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5816.891687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5817.015284] Lustre: 219662:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5818.865898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5819.069940] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5822.873215] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5824.356843] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5824.436182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5825.846111] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5825.917429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5828.628715] Lustre: server umount lustre-OST0000 complete [ 5833.184087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5833.187601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5833.193104] Lustre: Skipped 2 previous similar messages [ 5833.195699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5833.198485] Lustre: Skipped 3 previous similar messages [ 5839.355364] Lustre: server umount lustre-MDT0000 complete [ 5840.571469] LustreError: 217727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574021 with bad export cookie 9408785112237101918 [ 5840.574886] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5840.575683] LustreError: 217727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5851.156722] Lustre: server umount lustre-MDT0001 complete [ 5853.948049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.107222] LustreError: 221872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5854.114090] LustreError: 221872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5855.487638] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5858.289599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5859.825913] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5862.203885] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5864.039348] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5866.305994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5868.416564] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5870.914972] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5888.994958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5888.997133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.001446] Lustre: Skipped 2 previous similar messages [ 5893.778984] Lustre: server umount lustre-OST0000 complete [ 5899.233470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5899.235262] Lustre: Skipped 7 previous similar messages [ 5901.202302] Lustre: server umount lustre-MDT0000 complete [ 5902.429354] LustreError: 221855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574083 with bad export cookie 9408785112237102800 [ 5902.432122] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5902.435642] LustreError: 221855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5902.556586] Lustre: server umount lustre-MDT0001 complete [ 6003.836654] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6004.938625] Key type lgssc unregistered [ 6005.081465] LNet: 225555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6005.085368] LNetError: 225555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6005.095478] LNet: Removed LNI 192.168.202.152@tcp [ 6005.449115] Key type .llcrypt unregistered [ 6005.450297] Key type ._llcrypt unregistered [ 6013.548877] Key type ._llcrypt registered [ 6013.549962] Key type .llcrypt registered [ 6013.591027] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6019.686767] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6019.989694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6020.003470] alg: No test for adler32 (adler32-zlib) [ 6020.866915] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6020.965226] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6022.543126] Key type lgssc registered [ 6022.968978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6034.549415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6034.555303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.640388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6035.651928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6035.696638] Lustre: lustre-MDT0000: new disk, initializing [ 6035.718844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.725689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6037.057508] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6041.242915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6041.284236] Lustre: 228875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6041.312555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6041.315502] Lustre: Skipped 1 previous similar message [ 6041.318888] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6041.356922] Lustre: lustre-MDT0001: new disk, initializing [ 6041.388817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6041.401080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6041.403868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6042.755082] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6046.408395] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6048.255956] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6050.463320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6050.580750] Lustre: lustre-OST0000: new disk, initializing [ 6050.582343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6050.584694] Lustre: 230844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6050.620208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6052.151844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6052.154775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6052.190921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6052.550617] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6056.175971] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6057.578045] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6057.655430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6059.024988] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6059.103783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6062.560983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6062.562049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6062.563865] LustreError: Skipped 1 previous similar message [ 6062.563903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6062.570384] Lustre: Skipped 1 previous similar message [ 6067.683130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6067.684938] Lustre: Skipped 2 previous similar messages [ 6072.801252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6072.803079] Lustre: Skipped 1 previous similar message [ 6074.335125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6074.395572] Lustre: server umount lustre-OST0000 complete [ 6077.407889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6077.410138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.415255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6081.753501] Lustre: server umount lustre-MDT0000 complete [ 6083.028490] LustreError: 228868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574263 with bad export cookie 16790374531551261260 [ 6083.033316] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6083.154317] Lustre: server umount lustre-MDT0001 complete [ 6089.448073] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6093.085712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.256260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6094.714330] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6097.625363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6097.758609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6099.233748] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6100.264345] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6102.563457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6104.704353] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6107.608385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6107.676299] Lustre: lustre-OST0001: new disk, initializing [ 6107.678102] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6107.680546] Lustre: 236080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6107.715423] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6107.717513] Lustre: Skipped 1 previous similar message [ 6109.705110] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6113.798907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6113.802747] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6113.842257] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6115.199755] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6117.780672] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6121.334595] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 22:11:40 (1788574300) [ 6123.487842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6123.491022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.494785] Lustre: Skipped 1 previous similar message [ 6123.496669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6123.498354] Lustre: Skipped 1 previous similar message [ 6128.150557] Lustre: server umount lustre-MDT0000 complete [ 6129.122069] LustreError: 235274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6129.127973] LustreError: 235274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6129.401339] LustreError: 234905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574310 with bad export cookie 16790374531551262142 [ 6129.404120] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6129.404556] LustreError: 234905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6129.531228] Lustre: server umount lustre-MDT0001 complete [ 6140.416949] Lustre: server umount lustre-OST0000 complete [ 6151.904097] Lustre: server umount lustre-OST0001 complete [ 6154.088585] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6157.547411] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6169.673431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.763311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6169.774906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6169.811467] Lustre: lustre-MDT0000: new disk, initializing [ 6169.832534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6169.838895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6171.217688] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6175.500719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.547078] Lustre: 240019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6175.670565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6177.126749] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6180.917855] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6182.755428] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6185.009817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6185.148283] Lustre: lustre-OST0000: new disk, initializing [ 6185.150198] Lustre: Skipped 1 previous similar message [ 6185.152311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6185.154795] Lustre: Skipped 2 previous similar messages [ 6185.157514] Lustre: 241988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6186.877527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6186.879932] Lustre: Skipped 1 previous similar message [ 6186.881796] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6186.917933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6187.162508] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6190.960949] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6192.386432] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6192.462970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6193.889485] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6193.966484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6196.695071] Lustre: server umount lustre-OST0000 complete [ 6201.311906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6201.314211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6201.317710] Lustre: Skipped 3 previous similar messages [ 6201.319475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6201.321552] Lustre: Skipped 3 previous similar messages [ 6207.481988] Lustre: server umount lustre-MDT0000 complete [ 6208.723670] LustreError: 240011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574389 with bad export cookie 16790374531551264263 [ 6208.727608] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6208.729762] LustreError: 240011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6219.287243] Lustre: server umount lustre-MDT0001 complete [ 6222.162050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6222.345776] LustreError: 244201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6222.373412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6222.375849] Lustre: Skipped 2 previous similar messages [ 6223.731549] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6226.576253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.067558] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6230.568084] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6232.411938] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6234.690671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6236.784753] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6239.162213] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6385.632815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6385.635437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.639195] Lustre: Skipped 1 previous similar message [ 6385.640966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6385.642654] Lustre: Skipped 4 previous similar messages [ 6396.383126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6396.438994] Lustre: server umount lustre-OST0000 complete [ 6400.992342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6400.995834] LustreError: Skipped 1 previous similar message [ 6400.997725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.002712] Lustre: Skipped 1 previous similar message [ 6412.255146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6412.318995] Lustre: server umount lustre-MDT0000 complete [ 6413.598504] LustreError: 244937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574594 with bad export cookie 16790374531551265145 [ 6413.601443] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6413.601940] LustreError: 244937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6413.714375] Lustre: server umount lustre-MDT0001 complete [ 6511.662131] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6512.727318] Key type lgssc unregistered [ 6512.869543] LNet: 247811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6512.872807] LNetError: 247811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6512.880491] LNet: Removed LNI 192.168.202.152@tcp [ 6513.236117] Key type .llcrypt unregistered [ 6513.237255] Key type ._llcrypt unregistered [ 6521.184677] Key type ._llcrypt registered [ 6521.185685] Key type .llcrypt registered [ 6521.220367] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6527.265996] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6527.583664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6527.618057] alg: No test for adler32 (adler32-zlib) [ 6528.463849] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6528.554455] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6530.127119] Key type lgssc registered [ 6530.547590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6545.350058] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6548.972924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6548.977823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6550.062145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6550.071509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6550.106395] Lustre: lustre-MDT0000: new disk, initializing [ 6550.126821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6550.132965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6551.519738] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6555.778497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6555.820259] Lustre: 251847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6555.842596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6555.844658] Lustre: Skipped 1 previous similar message [ 6555.880954] Lustre: lustre-MDT0001: new disk, initializing [ 6555.915560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6555.928119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6555.932687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6557.327797] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6559.620745] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6561.913794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.027608] Lustre: lustre-OST0000: new disk, initializing [ 6562.030317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6562.034285] Lustre: 253478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6562.069612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6564.150241] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6567.423899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6567.427305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6567.476150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6568.521918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6568.602790] Lustre: lustre-OST0001: new disk, initializing [ 6568.605776] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6568.610412] Lustre: 254348:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6568.647731] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6570.640882] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6574.590807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6574.597620] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6574.628029] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6576.214276] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6580.935060] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6584.800996] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.804333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6584.804571] Lustre: Skipped 3 previous similar messages [ 6584.806311] Lustre: Skipped 2 previous similar messages [ 6589.922132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.924247] Lustre: Skipped 2 previous similar messages [ 6590.195861] Lustre: server umount lustre-MDT0000 complete [ 6591.475440] LustreError: 251838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574772 with bad export cookie 9514420253018534431 [ 6591.479766] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6591.480075] LustreError: 251838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6591.600151] Lustre: server umount lustre-MDT0001 complete [ 6603.269499] Lustre: server umount lustre-OST0000 complete [ 6614.745071] Lustre: server umount lustre-OST0001 complete [ 6619.394366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6622.666674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6625.653771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6628.868898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6634.966666] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6638.229914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6638.235221] Lustre: lustre-MDT0000: reset Object Index mappings [ 6638.379406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.695725] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6642.376990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6642.382955] Lustre: lustre-MDT0001: reset Object Index mappings [ 6643.772535] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6644.812831] Lustre: 260486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6646.995121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.150937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.153625] Lustre: Skipped 1 previous similar message [ 6649.056269] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6649.187295] LustreError: 260838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6649.189592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6649.194250] LustreError: 260838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6652.013401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6653.170906] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6654.277644] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6657.505473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6658.657299] Lustre: 262356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6663.136096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6663.138476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6663.142994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6663.144585] Lustre: Skipped 1 previous similar message [ 6665.947611] Lustre: server umount lustre-MDT0000 complete [ 6667.166249] LustreError: 259328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574847 with bad export cookie 9514420253018541165 [ 6667.168860] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6667.170681] LustreError: 259328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6667.283480] Lustre: server umount lustre-MDT0001 complete [ 6678.988121] Lustre: server umount lustre-OST0000 complete [ 6690.591796] Lustre: server umount lustre-OST0001 complete [ 6696.903823] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 22:21:15 (1788574875) [ 6701.775557] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6705.103763] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6717.153050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.232720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6717.242753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6717.277854] Lustre: lustre-MDT0000: new disk, initializing [ 6717.297973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6717.300070] Lustre: Skipped 1 previous similar message [ 6717.305220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6718.639485] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6722.847150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.889560] Lustre: 266603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6723.003451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6724.452651] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6728.262364] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6730.196586] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6732.567795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6732.683450] Lustre: lustre-OST0000: new disk, initializing [ 6732.685786] Lustre: Skipped 1 previous similar message [ 6732.688497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6732.691989] Lustre: Skipped 2 previous similar messages [ 6732.695318] Lustre: 268572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6733.881919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6733.884547] Lustre: Skipped 1 previous similar message [ 6733.886528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6733.922289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6734.759357] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6738.569626] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6740.071897] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6740.157880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6741.602409] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6741.687636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6744.036601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6744.037400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6744.038465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6744.038469] Lustre: Skipped 3 previous similar messages [ 6744.045501] Lustre: Skipped 4 previous similar messages [ 6744.467635] Lustre: server umount lustre-OST0000 complete [ 6748.640425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6748.643081] LustreError: Skipped 1 previous similar message [ 6748.644781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6748.649047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6748.650818] Lustre: Skipped 1 previous similar message [ 6751.899082] Lustre: server umount lustre-MDT0000 complete [ 6753.216241] LustreError: 268571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574933 with bad export cookie 9514420253018543237 [ 6753.219018] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6753.219815] LustreError: 268571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6753.338218] Lustre: server umount lustre-MDT0001 complete [ 6759.569444] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6762.856725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6762.969702] LustreError: 271502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6762.980321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6762.983785] Lustre: Skipped 2 previous similar messages [ 6764.197757] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6766.786399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.079294] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6768.992117] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6771.048361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6773.003301] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6775.997049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6776.051351] Lustre: lustre-OST0001: new disk, initializing [ 6776.052854] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6776.055038] Lustre: 273808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6777.660433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6777.664246] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6777.696684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6777.964547] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6782.363498] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6783.559058] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6785.180065] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6785.196355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6788.065201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.068031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6788.070310] Lustre: Skipped 4 previous similar messages [ 6788.071839] Lustre: Skipped 4 previous similar messages [ 6794.010750] Lustre: server umount lustre-MDT0000 complete [ 6795.177213] LustreError: 271485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574975 with bad export cookie 9514420253018544119 [ 6795.179910] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6795.181549] LustreError: 271485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6795.295085] Lustre: server umount lustre-MDT0001 complete [ 6806.542796] Lustre: server umount lustre-OST0000 complete [ 6818.082871] Lustre: server umount lustre-OST0001 complete [ 6823.925024] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6827.497363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6827.628541] LustreError: 277148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6827.643978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6827.646010] Lustre: Skipped 3 previous similar messages [ 6829.024561] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6831.827488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.398059] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6834.358600] Lustre: 278291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6836.692753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6837.859574] LustreError: 278644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6837.877673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6838.928983] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6841.884104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6844.037586] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6847.234935] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6847.467261] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6848.458285] Lustre: 280168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6850.256285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6850.263743] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6854.258052] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:23:53 (1788575033) [ 6867.938129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.939770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6867.940700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6867.940704] Lustre: Skipped 5 previous similar messages [ 6867.942735] Lustre: Skipped 3 previous similar messages [ 6871.788745] Lustre: server umount lustre-MDT0000 complete [ 6872.947048] LustreError: 277130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575053 with bad export cookie 9514420253018550489 [ 6872.950096] LustreError: 277130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6872.950577] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6873.056126] LustreError: 277176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6873.061397] LustreError: 277176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6882.271938] LustreError: 277143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6882.276711] LustreError: 277143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6887.391156] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6887.478678] Lustre: server umount lustre-MDT0001 complete [ 6903.263169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6903.297989] Lustre: server umount lustre-OST0000 complete [ 6910.549769] Lustre: server umount lustre-OST0001 complete [ 6912.634432] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6915.906306] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6930.360675] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6933.652329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6933.732753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6933.743107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6933.777435] Lustre: lustre-MDT0000: new disk, initializing [ 6933.804581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6935.119667] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6939.120253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6939.160530] Lustre: 284687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6939.262333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6940.545058] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6944.894175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6944.998644] Lustre: 286318:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6945.880720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6946.796739] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6950.860297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6952.763403] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6952.799881] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6957.056034] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6961.868610] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 22:25:40 (1788575140) [ 6963.169552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6963.172221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6963.172818] Lustre: Skipped 5 previous similar messages [ 6963.175073] Lustre: Skipped 13 previous similar messages [ 6968.671569] Lustre: server umount lustre-MDT0000 complete [ 6969.886781] LustreError: 284679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575150 with bad export cookie 9514420253018558448 [ 6969.890533] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6969.891501] LustreError: 284679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6970.001595] Lustre: server umount lustre-MDT0001 complete [ 6981.644643] Lustre: server umount lustre-OST0000 complete [ 6993.246657] Lustre: server umount lustre-OST0001 complete [ 6996.164799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.306073] LustreError: 289430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.314340] LustreError: 289430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6996.335696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6996.337915] Lustre: Skipped 7 previous similar messages [ 6997.643528] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7000.267308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7001.584979] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7003.780463] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7005.463497] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7007.568345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7009.446732] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7011.651989] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7028.084850] loop1: detected capacity change from 0 to 204800000 [ 7028.178535] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7028.182329] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7028.187495] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7029.824526] loop1: detected capacity change from 0 to 204800000 [ 7031.731839] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.765532] LustreError: 289419:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs23855: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7031.770471] LustreError: 289419:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs23855 [ 7031.772668] LustreError: 289419:0:(mgs_handler.c:517:mgs_target_reg()) fs23855-MDT0001: Error checking target index: rc = -13 [ 7031.776125] LustreError: fs23855-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7031.780165] LustreError: 293169:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 7031.782464] LustreError: 293169:0:(tgt_mount.c:2132:server_put_super()) no obd fs23855-MDT0001 [ 7031.784700] LustreError: 293169:0:(tgt_mount.c:116:server_deregister_mount()) fs23855-MDT0001 not registered [ 7031.798539] Lustre: server umount fs23855-MDT0001 complete [ 7031.801035] LustreError: 293169:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7032.291432] LustreError: 291260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.297855] LustreError: 291260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7034.516452] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.543527] Lustre: Setting parameter fs23855-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs23855-MDT0001 [ 7034.546535] Lustre: Skipped 2 previous similar messages [ 7034.560710] Lustre: srv-fs23855-MDT0001: No data found on store. Initialize space. [ 7034.563230] Lustre: Skipped 4 previous similar messages [ 7034.599631] Lustre: fs23855-MDT0001: new disk, initializing [ 7034.600974] Lustre: Skipped 3 previous similar messages [ 7034.632476] Lustre: srv-fs23855-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7034.634602] Lustre: 293633:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23855-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7034.636829] LustreError: 293633:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23855-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7035.914901] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7038.878222] Lustre: server umount fs23855-MDT0001 complete [ 7043.422279] loop1: detected capacity change from 0 to 204800000 [ 7045.382882] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7045.501214] Lustre: 295198:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23855-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7045.504068] LustreError: 295198:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23855-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7046.967449] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7048.637972] Lustre: server umount fs23855-MDT0001 complete [ 7052.773173] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.774359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7052.777394] Lustre: Skipped 1 previous similar message [ 7052.781180] Lustre: Skipped 6 previous similar messages [ 7058.203222] Lustre: server umount lustre-MDT0000 complete [ 7061.084444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7061.129335] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7062.649925] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7065.336881] loop1: detected capacity change from 0 to 204800000 [ 7066.598587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7067.267424] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7067.391661] Lustre: 297639:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23855-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7067.394928] LustreError: 297639:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23855-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7068.811441] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7070.432464] Lustre: server umount fs23855-MDT0001 complete [ 7077.344030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7083.282822] Lustre: server umount lustre-OST0000 complete [ 7087.073251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7087.075967] LustreError: Skipped 1 previous similar message [ 7090.716657] Lustre: server umount lustre-MDT0000 complete [ 7092.028429] LustreError: 289411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575272 with bad export cookie 9514420253018561122 [ 7092.031986] LustreError: 289411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7092.165440] Lustre: server umount lustre-MDT0001 complete [ 7095.605227] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7096.569930] Key type lgssc unregistered [ 7096.685412] LNet: 299977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7096.687559] LNetError: 299977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7096.695288] LNet: Removed LNI 192.168.202.152@tcp [ 7097.012145] Key type .llcrypt unregistered [ 7097.013391] Key type ._llcrypt unregistered [ 7104.410072] Key type ._llcrypt registered [ 7104.411059] Key type .llcrypt registered [ 7104.447100] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7104.854834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7104.939970] alg: No test for adler32 (adler32-zlib) [ 7105.796970] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7105.889632] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7107.471197] Key type lgssc registered [ 7107.875446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7110.668602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7110.674581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7111.851890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.216212] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7116.098899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7116.243995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7117.600622] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7120.052912] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7121.871177] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7124.126032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.286510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7125.347889] LustreError: 303047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7125.356399] LustreError: 303047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7126.358551] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7128.905963] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7130.595824] LustreError: 303047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.714925] LustreError: 303230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.523514] LustreError: 303230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.529407] LustreError: 303230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7144.635658] LustreError: 303049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.642901] LustreError: 303049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7145.748396] loop1: detected capacity change from 0 to 204800000 [ 7145.806330] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7145.810398] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7145.814889] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7147.590394] loop1: detected capacity change from 0 to 204800000 [ 7149.464758] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7149.513237] Lustre: fs20902-OST0000: new disk, initializing [ 7149.514739] Lustre: srv-fs20902-OST0000: No data found on store. Initialize space. [ 7149.517022] Lustre: 304975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7149.539505] Lustre: fs20902-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7151.537103] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7154.875926] LustreError: 303047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.882100] LustreError: 303047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7155.034666] Lustre: server umount fs20902-OST0000 complete [ 7159.241491] loop1: detected capacity change from 0 to 204800000 [ 7161.170263] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7161.233840] Lustre: fs20902-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7163.209677] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7166.184801] loop2: detected capacity change from 0 to 204800000 [ 7166.258790] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7166.264137] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7166.268418] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7167.969923] loop2: detected capacity change from 0 to 204800000 [ 7169.757474] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7169.787708] LustreError: fs20902-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7169.791412] LustreError: fs20902-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7169.794181] LustreError: 308012:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 7169.796395] LustreError: 308012:0:(tgt_mount.c:2132:server_put_super()) no obd fs20902-OST0001 [ 7169.798151] LustreError: 308012:0:(tgt_mount.c:116:server_deregister_mount()) fs20902-OST0001 not registered [ 7169.807698] Lustre: server umount fs20902-OST0001 complete [ 7169.809051] LustreError: 308012:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7172.580826] LustreError: 305408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7172.586099] LustreError: 305408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7173.208173] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7173.255997] Lustre: fs20902-OST0001: new disk, initializing [ 7173.259561] Lustre: srv-fs20902-OST0001: No data found on store. Initialize space. [ 7173.262522] Lustre: 308561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7173.285283] Lustre: fs20902-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7175.225799] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7178.662503] Lustre: server umount fs20902-OST0000 complete [ 7181.636484] Lustre: server umount fs20902-OST0001 complete [ 7185.594552] loop1: detected capacity change from 0 to 204800000 [ 7187.437704] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7189.429512] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7191.650442] loop2: detected capacity change from 0 to 204800000 [ 7193.569874] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7193.632210] Lustre: fs20902-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7193.635404] Lustre: Skipped 1 previous similar message [ 7195.616637] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7198.718386] Lustre: server umount fs20902-OST0000 complete [ 7201.821977] Lustre: server umount fs20902-OST0001 complete [ 7206.075234] LustreError: 305408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.080488] LustreError: 305408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 7207.391794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7207.394818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7207.399996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7208.419902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7208.420039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7208.420399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7213.457646] Lustre: server umount lustre-OST0000 complete [ 7218.657715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7218.658077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7218.658993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7218.659040] Lustre: Skipped 1 previous similar message [ 7218.667912] Lustre: Skipped 1 previous similar message [ 7220.893467] Lustre: server umount lustre-MDT0000 complete [ 7222.110294] LustreError: 303072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575402 with bad export cookie 16572682271874720670 [ 7222.113307] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7222.113606] LustreError: 303072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7222.232783] Lustre: server umount lustre-MDT0001 complete [ 7225.852751] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7226.917419] Key type lgssc unregistered [ 7227.047634] LNet: 314470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7227.050878] LNetError: 314470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7227.060560] LNet: Removed LNI 192.168.202.152@tcp [ 7227.417150] Key type .llcrypt unregistered [ 7227.418754] Key type ._llcrypt unregistered [ 7237.018083] Key type ._llcrypt registered [ 7237.019025] Key type .llcrypt registered [ 7237.051951] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 7243.167856] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7243.551818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7243.566078] alg: No test for adler32 (adler32-zlib) [ 7244.407703] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7244.489179] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7246.063193] Key type lgssc registered [ 7246.414581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7258.107506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7258.113418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.204800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7259.215433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7259.256354] Lustre: lustre-MDT0000: new disk, initializing [ 7259.279866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.286668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7260.676652] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7264.756953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7264.794483] Lustre: 318296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7264.816271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7264.817982] Lustre: Skipped 1 previous similar message [ 7264.849170] Lustre: lustre-MDT0001: new disk, initializing [ 7264.878352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7264.888793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7264.891730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7266.119902] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7269.663372] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7271.394836] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7273.477637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7273.568424] Lustre: lustre-OST0000: new disk, initializing [ 7273.569949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7273.572162] Lustre: 320264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7273.594953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7274.424238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7274.427034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7274.454599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7275.357088] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7278.849423] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7279.505872] Lustre: Mounted lustre-client - version 2.17.58_2_gfa0a981 [ 7279.812116] Lustre: Unmounted lustre-client [ 7284.692304] Lustre: server umount lustre-OST0000 complete [ 7289.825259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7289.826263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7289.828074] Lustre: Skipped 1 previous similar message [ 7294.944185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7294.946978] Lustre: Skipped 2 previous similar messages [ 7295.998486] Lustre: server umount lustre-MDT0000 complete [ 7297.270279] LustreError: 318289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575478 with bad export cookie 9289350063040135665 [ 7297.273429] LustreError: 318289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7297.273563] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7307.799454] Lustre: server umount lustre-MDT0001 complete [ 7311.551438] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7312.604117] Key type lgssc unregistered [ 7337.311357] LNet: 322419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7337.314037] LNetError: 322419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7411.045371] LNet: Removed LNI 192.168.202.152@tcp [ 7411.368907] Key type .llcrypt unregistered [ 7411.370649] Key type ._llcrypt unregistered [ 7417.562404] Key type ._llcrypt registered [ 7417.564030] Key type .llcrypt registered [ 7417.604840] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 7424.260136] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7424.571930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7424.620890] alg: No test for adler32 (adler32-zlib) [ 7425.490704] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7425.587420] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7427.175172] Key type lgssc registered [ 7427.568557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7438.094940] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:33:36 (1788575616) [ 7440.507253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7440.512873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.598350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7441.609058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7441.644127] Lustre: lustre-MDT0000: new disk, initializing [ 7441.670489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.678736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7443.082640] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7447.555442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.609805] Lustre: 325614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7447.637337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7447.640022] Lustre: Skipped 1 previous similar message [ 7447.681916] Lustre: lustre-MDT0001: new disk, initializing [ 7447.725878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7447.744516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7447.748264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7449.265176] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7453.407225] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7455.439656] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7457.859571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7457.967906] Lustre: lustre-OST0000: new disk, initializing [ 7457.970303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7457.974265] Lustre: 327583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7458.010887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.274254] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7464.081471] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7464.954132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7464.957409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7464.989092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7475.168540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7475.170865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.171657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7475.174138] Lustre: Skipped 1 previous similar message [ 7480.291480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7480.294877] Lustre: Skipped 2 previous similar messages [ 7481.242656] Lustre: server umount lustre-OST0000 complete [ 7483.872304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7483.876673] LustreError: Skipped 1 previous similar message [ 7483.879461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7483.885947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7485.409149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7488.669468] Lustre: server umount lustre-MDT0000 complete [ 7490.079233] LustreError: 326410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575670 with bad export cookie 4239195161632893560 [ 7490.082341] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7490.084482] LustreError: 326410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7490.204551] Lustre: server umount lustre-MDT0001 complete [ 7494.002370] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7495.105389] Key type lgssc unregistered [ 7495.232403] LNet: 329492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7495.235877] LNetError: 329492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7495.245292] LNet: Removed LNI 192.168.202.152@tcp [ 7495.599240] Key type .llcrypt unregistered [ 7495.600615] Key type ._llcrypt unregistered [ 7504.323658] Key type ._llcrypt registered [ 7504.325156] Key type .llcrypt registered [ 7504.376587] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7504.821331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7504.865293] alg: No test for adler32 (adler32-zlib) [ 7505.728649] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7505.831446] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7507.415248] Key type lgssc registered [ 7507.818724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7511.354522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7511.362039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7512.523887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7514.907600] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7518.033973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7518.182986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7519.719368] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7522.355671] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7524.222960] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7526.997369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7527.178729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.851688] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7534.254407] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7544.032337] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7544.571907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7546.049127] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 7547.872982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.874434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7547.878363] Lustre: Skipped 2 previous similar messages [ 7547.880518] Lustre: Skipped 1 previous similar message [ 7551.164248] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 7552.171713] Lustre: server umount lustre-MDT0000 complete [ 7552.992412] LustreError: 330738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7553.000531] LustreError: 330738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7553.571231] LustreError: 332577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575734 with bad export cookie 16128962229914897043 [ 7553.573891] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7553.576246] LustreError: 332577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7556.283119] LustreError: 330734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.292271] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [ 7558.112453] LustreError: 330732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7558.112670] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7559.734218] Lustre: server umount lustre-MDT0001 complete [ 7561.409388] Lustre: lustre-OST0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 7561.411133] Lustre: Skipped 1 previous similar message [ 7567.201109] Lustre: server umount lustre-OST0000 complete [ 7570.342084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.765534] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7572.766955] Lustre: 334912:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.152@tcp [ 7573.499748] Lustre: server umount MGS complete [ 7576.643159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.800646] LustreError: 335481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7576.824639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.227610] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7581.055664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.172888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7582.563649] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7585.387071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7587.335768] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7589.608190] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7593.167608] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 7594.960156] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7595.520791] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7601.121254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7601.122781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.123884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7601.126501] Lustre: Skipped 1 previous similar message [ 7601.128270] Lustre: Skipped 1 previous similar message [ 7603.339872] Lustre: server umount lustre-OST0000 complete [ 7606.240639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.714426] Lustre: server umount lustre-MDT0000 complete [ 7611.360191] LustreError: 335481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7611.365163] LustreError: 335481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7611.987973] LustreError: 335462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575792 with bad export cookie 16128962229914897869 [ 7611.990486] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7611.991139] LustreError: 335462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7612.099073] Lustre: server umount lustre-MDT0001 complete [ 7615.686049] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7616.696629] Key type lgssc unregistered [ 7616.829422] LNet: 338915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7616.832107] LNetError: 338915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7616.842336] LNet: Removed LNI 192.168.202.152@tcp [ 7617.172677] Key type .llcrypt unregistered [ 7617.173597] Key type ._llcrypt unregistered [ 7622.957292] Key type ._llcrypt registered [ 7622.958421] Key type .llcrypt registered [ 7622.995679] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 7629.466033] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7629.986705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7629.995770] alg: No test for adler32 (adler32-zlib) [ 7630.861884] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7630.959951] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7632.543175] Key type lgssc registered [ 7632.941776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7644.951799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7644.960968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.042780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7646.056362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7646.092929] Lustre: lustre-MDT0000: new disk, initializing [ 7646.117474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.125447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7647.608163] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7652.009826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7652.051658] Lustre: 342044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7652.074328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7652.076584] Lustre: Skipped 1 previous similar message [ 7652.117689] Lustre: lustre-MDT0001: new disk, initializing [ 7652.152626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7652.167323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7652.170949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7653.560868] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7657.618173] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7659.729286] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7662.443191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7662.586061] Lustre: lustre-OST0000: new disk, initializing [ 7662.587961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7662.591439] Lustre: 344012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7662.632519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7665.243914] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7668.765336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7668.772589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7668.829239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7669.498498] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7671.201328] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7671.270865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7672.734434] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7672.820174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7673.825230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7673.825309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.827340] LustreError: Skipped 1 previous similar message [ 7673.831401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7673.835040] Lustre: Skipped 1 previous similar message [ 7675.759205] Lustre: server umount lustre-OST0000 complete [ 7677.919961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7677.923306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.929028] Lustre: Skipped 1 previous similar message [ 7677.931328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7678.944312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7678.945565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.439456] Lustre: server umount lustre-MDT0000 complete [ 7684.978835] LustreError: 342036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575865 with bad export cookie 4945703758273311385 [ 7684.981797] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7684.983432] LustreError: 342036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7685.113373] Lustre: server umount lustre-MDT0001 complete [ 7688.171392] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:37:47 (1788575867) [ 7694.375434] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 7698.172867] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7711.651300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.758410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7711.773840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7711.813658] Lustre: lustre-MDT0000: new disk, initializing [ 7711.838941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7711.845071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7713.393908] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7718.019163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.073213] Lustre: 348759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7718.101487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7718.105043] Lustre: Skipped 1 previous similar message [ 7718.156846] Lustre: lustre-MDT0001: new disk, initializing [ 7718.200030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7718.214425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7718.220802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7719.987227] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7725.117974] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7727.853352] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7730.986988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7731.117262] Lustre: lustre-OST0000: new disk, initializing [ 7731.119516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7731.123201] Lustre: 350731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7731.159103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7733.624925] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7736.324973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7736.330849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7736.368575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7737.904095] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7751.648885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7751.652949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.654576] LustreError: Skipped 1 previous similar message [ 7751.655142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7751.657994] Lustre: Skipped 1 previous similar message [ 7751.659928] Lustre: Skipped 1 previous similar message [ 7753.756766] Lustre: server umount lustre-OST0000 complete [ 7756.770857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.772490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.779579] Lustre: Skipped 1 previous similar message [ 7761.303875] Lustre: server umount lustre-MDT0000 complete [ 7761.888037] LustreError: 349646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.893450] LustreError: 349646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7762.565364] LustreError: 348752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575943 with bad export cookie 4945703758273312253 [ 7762.567646] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.568737] LustreError: 348752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7762.722243] Lustre: server umount lustre-MDT0001 complete [ 7766.569574] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7767.727526] Key type lgssc unregistered [ 7767.859442] LNet: 352879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7767.862279] LNetError: 352879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7767.875361] LNet: Removed LNI 192.168.202.152@tcp [ 7768.203492] Key type .llcrypt unregistered [ 7768.205105] Key type ._llcrypt unregistered [ 7805.509078] Key type ._llcrypt registered [ 7805.510376] Key type .llcrypt registered [ 7805.550994] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7805.664225] Key type .llcrypt unregistered [ 7805.665654] Key type ._llcrypt unregistered [ 7837.464972] Key type ._llcrypt registered [ 7837.465897] Key type .llcrypt registered [ 7837.501599] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7837.605480] Key type .llcrypt unregistered [ 7837.606639] Key type ._llcrypt unregistered [ 7851.434944] Key type ._llcrypt registered [ 7851.436513] Key type .llcrypt registered [ 7851.495790] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7851.622326] Key type .llcrypt unregistered [ 7851.623938] Key type ._llcrypt unregistered [ 7865.014894] Key type ._llcrypt registered [ 7865.016595] Key type .llcrypt registered [ 7865.066109] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7865.182290] Key type .llcrypt unregistered [ 7865.183947] Key type ._llcrypt unregistered [ 7903.695626] Key type ._llcrypt registered [ 7903.697792] Key type .llcrypt registered [ 7903.767790] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7903.941955] Key type .llcrypt unregistered [ 7903.943873] Key type ._llcrypt unregistered [ 7921.318829] Key type ._llcrypt registered [ 7921.320739] Key type .llcrypt registered [ 7921.376147] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7921.522360] Key type .llcrypt unregistered [ 7921.523901] Key type ._llcrypt unregistered [ 7939.047785] Key type ._llcrypt registered [ 7939.049652] Key type .llcrypt registered [ 7939.107472] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7939.265291] Key type .llcrypt unregistered [ 7939.267104] Key type ._llcrypt unregistered [ 7957.383764] Key type ._llcrypt registered [ 7957.388394] Key type .llcrypt registered [ 7957.458974] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7957.623286] Key type .llcrypt unregistered [ 7957.625264] Key type ._llcrypt unregistered [ 7980.786340] Key type ._llcrypt registered [ 7980.788385] Key type .llcrypt registered [ 7980.894266] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7981.136734] Key type .llcrypt unregistered [ 7981.138596] Key type ._llcrypt unregistered [ 8005.331983] Key type ._llcrypt registered [ 8005.333186] Key type .llcrypt registered [ 8005.376442] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8005.524147] Key type .llcrypt unregistered [ 8005.525519] Key type ._llcrypt unregistered [ 8081.164714] Key type ._llcrypt registered [ 8081.170864] Key type .llcrypt registered [ 8081.347705] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8081.801423] Key type .llcrypt unregistered [ 8081.802950] Key type ._llcrypt unregistered [ 8129.730670] Key type ._llcrypt registered [ 8129.732625] Key type .llcrypt registered [ 8129.827152] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8130.055275] Key type .llcrypt unregistered [ 8130.057805] Key type ._llcrypt unregistered [ 8167.039938] Key type ._llcrypt registered [ 8167.041474] Key type .llcrypt registered [ 8167.119541] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8167.318762] Key type .llcrypt unregistered [ 8167.320810] Key type ._llcrypt unregistered [ 8219.942807] Key type ._llcrypt registered [ 8219.944886] Key type .llcrypt registered [ 8220.019439] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8236.245853] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8237.593714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8237.643393] alg: No test for adler32 (adler32-zlib) [ 8238.861690] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 8239.150484] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 8240.807253] Key type lgssc registered [ 8242.255085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8283.067916] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8283.089428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.377971] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8284.413296] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8284.502360] Lustre: tst250fs-MDT0000: new disk, initializing [ 8284.609149] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.637701] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8290.009327] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8301.675595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8301.821822] Lustre: 372882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8301.863720] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8301.866146] Lustre: Skipped 1 previous similar message [ 8301.916191] Lustre: tst250fs-MDT0001: new disk, initializing [ 8302.046783] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8302.083525] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8302.100146] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8307.103930] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8312.411408] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8320.125417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8320.459750] Lustre: tst250fs-OST0000: new disk, initializing [ 8320.473612] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8320.486365] Lustre: 374514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8320.487973] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [ 8320.662317] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.212634] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8326.227415] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8326.272745] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 8327.921675] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8339.980617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8340.294512] Lustre: tst250fs-OST0001: new disk, initializing [ 8340.300837] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8340.311549] Lustre: 375382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8340.528711] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8347.973991] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8350.393736] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8350.410132] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8350.556172] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8354.136094] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8356.754467] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [ 8356.767284] Lustre: Skipped 1 previous similar message [ 8381.419805] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8381.450851] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8381.453272] Lustre: Skipped 3 previous similar messages [ 8381.470940] Lustre: Skipped 3 previous similar messages [ 8386.531695] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8386.562763] Lustre: Skipped 3 previous similar messages [ 8388.032685] Lustre: server umount tst250fs-MDT0000 complete [ 8394.537606] LustreError: 372873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576575 with bad export cookie 13454585351758683627 [ 8394.543951] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8394.551897] LustreError: 372873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8395.083428] Lustre: server umount tst250fs-MDT0001 complete [ 8410.905937] Lustre: server umount tst250fs-OST0000 complete [ 8413.025337] Lustre: 371103:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788576577/real 1788576577] req@ffff972606d9d500 x1875458307201280/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788576593 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8413.070217] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.682962] Lustre: server umount tst250fs-OST0001 complete [ 8441.314371] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8452.465701] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8490.287662] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:51:08 (1788576668) [ 8492.555766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8513.081172] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8525.795753] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8563.176226] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8219 sec ========== 22:52:20 (1788576740) [ 8565.531978] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:52:22 (1788576742) === [ 8570.004866] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:52:27 (1788576747) === [ 8618.032542] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8621.266987] Key type lgssc unregistered [ 8621.526639] LNet: 384532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8621.536677] LNetError: 384532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8621.554079] LNet: Removed LNI 192.168.202.152@tcp [ 8622.776160] Key type .llcrypt unregistered [ 8622.778355] Key type ._llcrypt unregistered