[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 522863887 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002311] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006892] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009123] LSM: Security Framework initializing [ 0.010041] Yama: becoming mindful. [ 0.011023] SELinux: Initializing. [ 0.012041] *** VALIDATE selinux *** [ 0.020450] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025051] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027107] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030024] *** VALIDATE tmpfs *** [ 0.031422] *** VALIDATE proc *** [ 0.032216] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041064] debug: unmapping init [mem 0xffffffffb3459000-0xffffffffb3460fff] [ 0.043835] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047009] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052265] rcu: Hierarchical SRCU implementation. [ 0.054470] smp: Bringing up secondary CPUs ... [ 0.055536] x86: Booting SMP configuration: [ 0.056018] .... node #0, CPUs: #1 #2 #3 [ 0.059477] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142732] node 0 deferred pages initialised in 77ms [ 0.144119] devtmpfs: initialized [ 0.145226] x86/mm: Memory block size: 128MB [ 0.147683] gcov: version magic: 0x41383552 [ 0.149031] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151251] pinctrl core: initialized pinctrl subsystem [ 0.152179] [ 0.152804] ************************************************************* [ 0.153016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154011] ** ** [ 0.155012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156011] ** ** [ 0.157012] ** This means that this kernel is built to expose internal ** [ 0.158011] ** IOMMU data structures, which may compromise security on ** [ 0.159012] ** your system. ** [ 0.160010] ** ** [ 0.161012] ** If you see this message and you are not debugging the ** [ 0.162012] ** kernel, report this immediately to your vendor! ** [ 0.163011] ** ** [ 0.164011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165011] ************************************************************* [ 0.166659] NET: Registered protocol family 16 [ 0.167434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170510] cpuidle: using governor menu [ 0.171487] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174494] PCI: Using configuration type 1 for base access [ 0.176121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186073] cryptd: max_cpu_qlen set to 1000 [ 0.188231] ACPI: Added _OSI(Module Device) [ 0.189011] ACPI: Added _OSI(Processor Device) [ 0.190013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194988] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197568] ACPI: Interpreter enabled [ 0.198052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.199009] ACPI: Using IOAPIC for interrupt routing [ 0.200108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.201378] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.210862] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.211038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.212019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.213072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.215296] acpiphp: Slot [2] registered [ 0.216134] acpiphp: Slot [5] registered [ 0.217105] acpiphp: Slot [6] registered [ 0.218110] acpiphp: Slot [7] registered [ 0.219106] acpiphp: Slot [8] registered [ 0.220133] acpiphp: Slot [9] registered [ 0.221113] acpiphp: Slot [10] registered [ 0.223107] acpiphp: Slot [3] registered [ 0.224134] acpiphp: Slot [4] registered [ 0.225133] acpiphp: Slot [11] registered [ 0.226070] acpiphp: Slot [12] registered [ 0.227090] acpiphp: Slot [13] registered [ 0.228056] acpiphp: Slot [14] registered [ 0.229053] acpiphp: Slot [15] registered [ 0.230034] acpiphp: Slot [16] registered [ 0.231062] acpiphp: Slot [17] registered [ 0.233095] acpiphp: Slot [18] registered [ 0.235107] acpiphp: Slot [19] registered [ 0.236054] acpiphp: Slot [20] registered [ 0.237054] acpiphp: Slot [21] registered [ 0.238039] acpiphp: Slot [22] registered [ 0.239053] acpiphp: Slot [23] registered [ 0.240069] acpiphp: Slot [24] registered [ 0.241059] acpiphp: Slot [25] registered [ 0.242046] acpiphp: Slot [26] registered [ 0.243052] acpiphp: Slot [27] registered [ 0.244053] acpiphp: Slot [28] registered [ 0.245000] acpiphp: Slot [29] registered [ 0.245000] acpiphp: Slot [30] registered [ 0.247094] acpiphp: Slot [31] registered [ 0.249103] PCI host bridge to bus 0000:00 [ 0.251027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.254023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.256020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.259022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.262022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.265025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.267134] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.269929] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.272470] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.283014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.289061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.293021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.295016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.297021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.300549] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.303836] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.306114] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.308792] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.314013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.326853] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.331014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.336719] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.345018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.361018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.379014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.388625] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.399015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.411015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.442019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.459472] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.473026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.483026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.520028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.569792] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.580017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.591014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.637018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.655486] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.697034] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.716024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.744016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.752321] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.770013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.788013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.819015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.831375] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.833364] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.836359] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.839343] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.841225] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.847047] iommu: Default domain type: Passthrough [ 0.848400] SCSI subsystem initialized [ 0.850110] ACPI: bus type USB registered [ 0.851119] usbcore: registered new interface driver usbfs [ 0.853107] usbcore: registered new interface driver hub [ 0.855073] usbcore: registered new device driver usb [ 0.857168] pps_core: LinuxPPS API ver. 1 registered [ 0.859010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.862058] PTP clock support registered [ 0.864068] EDAC MC: Ver: 3.0.0 [ 0.865517] PCI: Using ACPI for IRQ routing [ 0.866546] NetLabel: Initializing [ 0.867006] NetLabel: domain hash size = 128 [ 0.867923] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.868110] NetLabel: unlabeled traffic allowed by default [ 0.869127] vgaarb: loaded [ 0.872022] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.873008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.878218] clocksource: Switched to clocksource kvm-clock [ 0.977148] VFS: Disk quotas dquot_6.6.0 [ 0.978713] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.981644] *** VALIDATE ramfs *** [ 0.983036] *** VALIDATE hugetlbfs *** [ 0.984707] pnp: PnP ACPI init [ 0.987031] pnp: PnP ACPI: found 6 devices [ 1.016840] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.020329] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.022537] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.024584] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.026828] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.029136] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.031843] NET: Registered protocol family 2 [ 1.034355] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.039289] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.042988] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.048376] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.051881] TCP: Hash tables configured (established 65536 bind 65536) [ 1.055041] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.058410] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.061529] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.064791] NET: Registered protocol family 1 [ 1.067199] RPC: Registered named UNIX socket transport module. [ 1.069307] RPC: Registered udp transport module. [ 1.071083] RPC: Registered tcp transport module. [ 1.072875] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.075437] NET: Registered protocol family 44 [ 1.077299] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.078691] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.080253] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.082508] PCI: CLS 0 bytes, default 64 [ 1.084703] Unpacking initramfs... [ 2.505786] debug: unmapping init [mem 0xffffa0ec7cc54000-0xffffa0ec7ffbffff] [ 2.509258] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.511018] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.513274] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.033723] Initialise system trusted keyrings [ 3.035735] Key type blacklist registered [ 3.037614] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.050889] zbud: loaded [ 3.055691] *** VALIDATE nfs *** [ 3.057148] *** VALIDATE nfs4 *** [ 3.059241] pstore: using deflate compression [ 3.065847] Platform Keyring initialized [ 3.186273] NET: Registered protocol family 38 [ 3.188065] Key type asymmetric registered [ 3.189673] Asymmetric key parser 'x509' registered [ 3.191511] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.196128] io scheduler mq-deadline registered [ 3.197560] io scheduler kyber registered [ 3.199184] io scheduler bfq registered [ 3.202116] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.204838] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.207532] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.210196] ACPI: Power Button [PWRF] [ 3.214875] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.221358] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.241471] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.252929] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.277470] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.303649] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.330455] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.335433] Non-volatile memory driver v1.3 [ 3.337408] Linux agpgart interface v0.103 [ 3.364907] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.368275] vda: detected capacity change from 0 to 74711040 [ 3.412810] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.416383] vdb: detected capacity change from 0 to 1073741824 [ 3.449666] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.453970] vdc: detected capacity change from 0 to 2621440000 [ 3.479600] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.483424] vdd: detected capacity change from 0 to 2621440000 [ 3.499570] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.502661] vde: detected capacity change from 0 to 4294967296 [ 3.528372] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.531424] vdf: detected capacity change from 0 to 4294967296 [ 3.547376] libphy: Fixed MDIO Bus: probed [ 3.565956] usbcore: registered new interface driver usbserial_generic [ 3.569185] usbserial: USB Serial support registered for generic [ 3.572138] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.576531] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.578583] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.581191] mousedev: PS/2 mouse device common for all mice [ 3.584272] rtc_cmos 00:05: RTC can wake from S4 [ 3.589176] rtc_cmos 00:05: registered as rtc0 [ 3.591734] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.595603] intel_pstate: CPU model not supported [ 3.596088] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.603142] hid: raw HID events driver (C) Jiri Kosina [ 3.608745] usbcore: registered new interface driver usbhid [ 3.611435] usbhid: USB HID core driver [ 3.613227] drop_monitor: Initializing network drop monitor service [ 3.615791] Initializing XFRM netlink socket [ 3.616275] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.617672] NET: Registered protocol family 10 [ 3.622121] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.622836] Segment Routing with IPv6 [ 3.626779] NET: Registered protocol family 17 [ 3.628853] mpls_gso: MPLS GSO support [ 3.634271] RAS: Correctable Errors collector initialized. [ 3.636021] AVX version of gcm_enc/dec engaged. [ 3.637656] AES CTR mode by8 optimization enabled [ 3.708930] sched_clock: Marking stable (3708907895, 0)->(4662308923, -953401028) [ 3.711929] registered taskstats version 1 [ 3.714393] Loading compiled-in X.509 certificates [ 3.716400] zswap: loaded using pool lzo/zbud [ 3.743314] Key type big_key registered [ 3.758844] Key type encrypted registered [ 3.760617] ima: No TPM chip found, activating TPM-bypass! [ 3.762382] ima: Allocated hash algorithm: sha1 [ 3.763871] ima: No architecture policies found [ 3.765289] evm: Initialising EVM extended attributes: [ 3.766872] evm: security.selinux [ 3.768134] evm: security.ima [ 3.769086] evm: security.capability [ 3.770208] evm: HMAC attrs: 0x1 [ 3.772338] rtc_cmos 00:05: setting system clock to 2026-07-30 12:54:10 UTC (1785416050) [ 3.777831] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 3.780411] debug: unmapping init [mem 0xffffffffb3182000-0xffffffffb3458fff] [ 3.793106] Write protecting the kernel read-only data: 28672k [ 3.796284] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.798725] debug: unmapping init [mem 0xffffffffb2114000-0xffffffffb21fffff] [ 3.831289] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.838093] systemd[1]: Detected virtualization kvm. [ 3.839683] systemd[1]: Detected architecture x86-64. [ 3.841074] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.866898] systemd[1]: No hostname configured. [ 3.868289] systemd[1]: Set hostname to . [ 3.869856] random: systemd: uninitialized urandom read (16 bytes read) [ 3.871732] systemd[1]: Initializing machine ID from random generator. [ 3.916646] random: ln: uninitialized urandom read (6 bytes read) [ 4.008964] random: systemd: uninitialized urandom read (16 bytes read) [ 4.012059] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.016459] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.021217] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.587619] device-mapper: uevent: version 1.0.3 [ 4.589990] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.925737] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.233019] virtio_net virtio0 ens2: renamed from eth0 [ 5.255829] scsi host0: ata_piix [ 5.264964] scsi host1: ata_piix [ 5.266817] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.279603] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.944211] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.277798] random: crng init done [ 10.279280] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.549654] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.593593] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.905147] SELinux: Disabled at runtime. [ 11.967600] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.974595] systemd[1]: Detected virtualization kvm. [ 11.975906] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.496168] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.498647] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.503437] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.506576] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.508828] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.514623] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.538147] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ 12.644710] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.984358] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.301887] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.308452] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.478378] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.488712] EDAC sbridge: Ver: 1.1.2 [ 14.929323] Key type dns_resolver registered [ 15.235584] NFS: Registering the id_resolver key type [ 15.237586] Key type id_resolver registered [ 15.239212] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg104-server login: [ 39.874557] libcfs: loading out-of-tree module taints kernel. [ 39.896526] Key type ._llcrypt registered [ 39.898095] Key type .llcrypt registered [ 39.940182] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 58.546035] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 61.017515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.048355] alg: No test for adler32 (adler32-zlib) [ 62.864798] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 64.169191] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 66.111178] Key type lgssc registered [ 68.415542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.996909] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 137.345329] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 154.159464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.228598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.739855] hrtimer: interrupt took 13761741 ns [ 155.545223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.648909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.766952] Lustre: lustre-MDT0000: new disk, initializing [ 155.927318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.965382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 162.285371] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.171640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.325589] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.378861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.394143] Lustre: Skipped 1 previous similar message [ 178.662757] Lustre: lustre-MDT0001: new disk, initializing [ 178.780713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.814551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.825524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.495768] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.257840] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.834490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.050826] Lustre: lustre-OST0000: new disk, initializing [ 202.053932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.058186] Lustre: 8454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.120161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.977418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 209.993732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.145357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.368974] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.387131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.791797] Lustre: lustre-OST0001: new disk, initializing [ 234.800274] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 234.818993] Lustre: 9528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.957758] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.792712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 242.800598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 242.964554] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.992297] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.513572] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.591814] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 285.233488] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing check_logdir /tmp/testlogs/ [ 290.444205] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing yml_node [ 295.694364] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 298.734676] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 302.298330] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 304.435264] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 30 08:59:08 EDT 2026 [ 326.581825] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 328.248829] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 345.568589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.571266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.578027] Lustre: Skipped 3 previous similar messages [ 345.602857] Lustre: Skipped 3 previous similar messages [ 350.689615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.708270] Lustre: Skipped 3 previous similar messages [ 355.808570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.818581] Lustre: Skipped 3 previous similar messages [ 358.368131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 358.983471] Lustre: server umount lustre-MDT0000 complete [ 364.932309] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785416411 with bad export cookie 6480898735416080417 [ 364.937691] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 364.945066] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 366.071966] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.075814] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.076179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.092013] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 366.181836] Lustre: Skipped 1 previous similar message [ 369.121533] LustreError: 10193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.153167] LustreError: 10193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 372.838547] Lustre: server umount lustre-MDT0001 complete [ 378.498619] Lustre: server umount lustre-OST0000 complete [ 385.582467] Lustre: server umount lustre-OST0001 complete [ 397.490724] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 414.882939] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 455.601418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.983443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 456.024967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 456.159698] Lustre: lustre-MDT0000: new disk, initializing [ 456.308114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.347645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 461.859384] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 474.088203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.213772] Lustre: 15126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 474.286717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 474.300766] Lustre: Skipped 1 previous similar message [ 474.407058] Lustre: lustre-MDT0001: new disk, initializing [ 474.516589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 474.555680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 474.577980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 479.899969] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 490.261340] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 498.118384] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 506.116736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 506.643602] Lustre: lustre-OST0000: new disk, initializing [ 506.646206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 506.649347] Lustre: 17094:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 506.715331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.711912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 508.729758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 508.871971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 515.423438] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 528.718165] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 534.709124] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 534.976501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 540.625851] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 540.918421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 544.761309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.761664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.776247] LustreError: Skipped 1 previous similar message [ 544.783659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.789395] Lustre: Skipped 1 previous similar message [ 544.822035] Lustre: Skipped 4 previous similar messages [ 549.841902] Lustre: server umount lustre-OST0000 complete [ 554.977853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.003238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.965817] Lustre: server umount lustre-MDT0000 complete [ 560.096755] LustreError: 15133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.132811] LustreError: 15133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 564.095535] LustreError: 15116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785416610 with bad export cookie 6480898735416081950 [ 564.101720] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.106932] LustreError: 15116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 564.636449] Lustre: server umount lustre-MDT0001 complete [ 572.044960] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:03:36 (1785416616) [ 596.051634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.535414] LustreError: 19971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.593117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.380260] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 602.083540] LustreError: 19972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.327073] LustreError: 19971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.341697] LustreError: 19971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 612.627180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.621751] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 628.907485] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 638.030110] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 641.600287] Lustre: Failing over lustre-MDT0000 [ 641.624469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.661360] Lustre: Skipped 1 previous similar message [ 641.692459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 641.702877] Lustre: Skipped 2 previous similar messages [ 642.163980] Lustre: server umount lustre-MDT0000 complete [ 646.635315] LustreError: 19971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.661262] LustreError: 19971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 662.950738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.007182] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785416693/real 1785416693] req@ffffa0ecfe02f100 x1872144483140736/t0(0) o400->MGC192.168.201.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785416709 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 663.049856] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.743393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 673.762342] Lustre: Skipped 1 previous similar message [ 673.895060] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 678.517718] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 678.885865] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 678.891275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 678.918901] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 691.649252] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 699.361320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.362070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 699.389081] Lustre: Skipped 1 previous similar message [ 699.398162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 699.416330] Lustre: Skipped 1 previous similar message [ 702.834219] Lustre: server umount lustre-MDT0000 complete [ 704.484375] LustreError: 19968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.532428] LustreError: 19968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 708.252174] LustreError: 19951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785416754 with bad export cookie 6480898735416083364 [ 708.270087] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 708.274177] LustreError: 19951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 708.889851] Lustre: server umount lustre-MDT0001 complete [ 719.911696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.361985] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 736.450439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.837753] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 752.605399] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 762.179775] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 766.954410] Lustre: Failing over lustre-MDT0000 [ 767.521416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 767.532841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.545988] Lustre: Skipped 1 previous similar message [ 767.945388] Lustre: server umount lustre-MDT0000 complete [ 770.539714] LustreError: 23314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 770.574747] LustreError: 23314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 786.914049] Lustre: 3650:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785416817/real 1785416817] req@ffffa0ecc47daa00 x1872144483173760/t0(0) o400->MGC192.168.201.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785416833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 786.969633] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 788.944236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.628995] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 802.761643] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 802.805413] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 802.815488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 802.827577] Lustre: Skipped 1 previous similar message [ 802.877127] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 815.139565] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 823.272091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 823.273702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.280418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 823.306055] Lustre: Skipped 2 previous similar messages [ 826.377599] Lustre: server umount lustre-MDT0000 complete [ 830.945330] LustreError: 24884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785416877 with bad export cookie 6480898735416084302 [ 830.953512] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 830.960309] LustreError: 24884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 831.365259] Lustre: server umount lustre-MDT0001 complete [ 843.108389] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:08:06 (1785416886) [ 863.760052] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 875.442127] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 934.905755] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 948.495252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.686961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 948.733363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 948.857909] Lustre: lustre-MDT0000: new disk, initializing [ 949.008784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 949.017698] Lustre: Skipped 3 previous similar messages [ 949.037451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 954.849877] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 968.201895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.410904] Lustre: 29717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 968.873380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 973.480711] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 985.662269] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 994.463050] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1008.940368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1010.092672] Lustre: 31688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1011.709530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1011.806989] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1011.959624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1011.994577] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1021.932434] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1038.121444] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1046.449329] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1078.240704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1078.248036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.266690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1078.269518] Lustre: Skipped 1 previous similar message [ 1083.576733] Lustre: server umount lustre-OST0000 complete [ 1095.019333] Lustre: server umount lustre-MDT0000 complete [ 1099.232523] LustreError: 32412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.246807] LustreError: 32412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 1099.684748] LustreError: 29709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785417146 with bad export cookie 6480898735416084708 [ 1099.690345] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.697773] LustreError: 29709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1100.484887] Lustre: server umount lustre-MDT0001 complete [ 1115.723602] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1120.004153] Key type lgssc unregistered [ 1120.498559] LNet: 33672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.507733] LNetError: 33672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1120.526198] LNet: Removed LNI 192.168.201.104@tcp [ 1121.621604] Key type .llcrypt unregistered [ 1121.627620] Key type ._llcrypt unregistered [ 1156.856394] Key type ._llcrypt registered [ 1156.859483] Key type .llcrypt registered [ 1156.977297] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1158.655131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1158.736132] alg: No test for adler32 (adler32-zlib) [ 1160.181293] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 1160.640379] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1162.415469] Key type lgssc registered [ 1164.186429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1175.672239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1175.708080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.790368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.486502] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1187.315832] LustreError: 35092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.350371] LustreError: 35092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1192.423775] LustreError: 35091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.544375] LustreError: 35092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.721299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.181799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1208.215862] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1220.439778] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1227.659602] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1236.333774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1236.956246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1244.163991] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1248.261932] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1249.360082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1259.381743] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1276.585888] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:15:20 (1785417320) [ 1278.650620] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1280.884680] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1289.223685] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:15:33 (1785417333) [ 1305.224418] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:15:47 (1785417347) [ 1306.512926] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1318.573543] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1318.580573] Lustre: Skipped 2 previous similar messages [ 1321.311769] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:16:05 (1785417365) [ 1324.038810] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1324.041788] Lustre: Skipped 2 previous similar messages [ 1332.871809] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1332.885888] Lustre: Skipped 1 previous similar message [ 1349.324275] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1349.328427] Lustre: Skipped 8 previous similar messages [ 1351.466854] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:16:35 (1785417395) [ 1355.160387] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1360.023634] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1370.381818] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:16:54 (1785417414) [ 1385.834525] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1385.837426] Lustre: Skipped 4 previous similar messages [ 1388.054116] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:17:12 (1785417432) [ 1407.988511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1407.998177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1408.007877] LustreError: Skipped 1 previous similar message [ 1408.012081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1408.033465] Lustre: Skipped 1 previous similar message [ 1408.074163] Lustre: Skipped 1 previous similar message [ 1413.095706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.116688] Lustre: Skipped 1 previous similar message [ 1418.213560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1418.221449] Lustre: Skipped 1 previous similar message [ 1420.255351] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1420.483251] Lustre: server umount lustre-OST0000 complete [ 1428.450397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.491220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.508293] Lustre: Skipped 1 previous similar message [ 1433.568712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.576676] Lustre: Skipped 1 previous similar message [ 1440.223483] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1440.458553] Lustre: server umount lustre-MDT0000 complete [ 1443.811993] LustreError: 35087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.829573] LustreError: 35087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1445.170391] LustreError: 35073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785417491 with bad export cookie 2777634463944331314 [ 1445.172585] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1445.183276] LustreError: 35073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1445.994679] Lustre: server umount lustre-MDT0001 complete [ 1460.432383] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1464.099334] Key type lgssc unregistered [ 1464.472375] LNet: 42961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.486717] LNetError: 42961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.519387] LNet: Removed LNI 192.168.201.104@tcp [ 1466.009631] Key type .llcrypt unregistered [ 1466.010940] Key type ._llcrypt unregistered [ 1490.180566] Key type ._llcrypt registered [ 1490.182719] Key type .llcrypt registered [ 1490.341228] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 1511.911551] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1513.546654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1513.731604] alg: No test for adler32 (adler32-zlib) [ 1515.103834] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 1515.441410] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1517.202437] Key type lgssc registered [ 1518.675977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1574.803149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1574.844873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1576.236502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1576.259307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1576.324543] Lustre: lustre-MDT0000: new disk, initializing [ 1576.367398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.376610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1581.796280] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1594.436332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.622804] Lustre: 46123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1594.739505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1594.746408] Lustre: Skipped 1 previous similar message [ 1594.853744] Lustre: lustre-MDT0001: new disk, initializing [ 1594.964908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1595.035643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1595.053850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1600.583410] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1610.673439] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1617.863859] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1627.524911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1628.169387] Lustre: lustre-OST0000: new disk, initializing [ 1628.178925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1628.244932] Lustre: 48092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1628.394966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.351516] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1638.012404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1638.022916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1638.141199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1648.517251] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1653.764463] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1653.961934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1658.643379] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1658.833231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1663.969805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.975380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.991790] Lustre: Skipped 1 previous similar message [ 1664.013022] Lustre: Skipped 1 previous similar message [ 1669.093523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1669.104471] Lustre: Skipped 1 previous similar message [ 1674.214781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1674.232363] Lustre: Skipped 1 previous similar message [ 1675.231287] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1675.491762] Lustre: server umount lustre-OST0000 complete [ 1681.888300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1681.890778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.904903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.450996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1686.717643] Lustre: server umount lustre-MDT0000 complete [ 1690.859260] LustreError: 46114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785417737 with bad export cookie 8631075828532049622 [ 1690.869569] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1690.878173] LustreError: 46114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1691.567286] Lustre: server umount lustre-MDT0001 complete [ 1703.918398] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:22:28 (1785417748) [ 1712.167097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.721616] LustreError: 50555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.748148] LustreError: 50555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1712.830930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.982760] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1718.241746] LustreError: 50556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.337873] LustreError: 50555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.459727] LustreError: 50556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.774343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.697780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1739.326116] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1748.502321] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1755.750209] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1764.593034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1764.981638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.298749] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1782.626861] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1788.325735] LustreError: 53086:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1790.024908] Lustre: Modifying parameter general.timeout=1 in log params [ 1791.391796] Lustre: Modifying parameter general.timeout=2 in log params [ 1794.513227] Lustre: Modifying parameter general.timeout=4 in log params [ 1794.516897] Lustre: Skipped 1 previous similar message [ 1798.814445] Lustre: Modifying parameter general.timeout=7 in log params [ 1798.823633] Lustre: Skipped 2 previous similar messages [ 1807.859888] Lustre: Modifying parameter general.timeout=15 in log params [ 1807.864678] Lustre: Skipped 7 previous similar messages [ 1824.239444] Lustre: Modifying parameter general.timeout=29 in log params [ 1824.244962] Lustre: Skipped 13 previous similar messages [ 1856.415298] Lustre: Modifying parameter general.timeout=56 in log params [ 1856.421021] Lustre: Skipped 26 previous similar messages [ 1921.592107] Lustre: Modifying parameter general.timeout=107 in log params [ 1921.602800] Lustre: Skipped 50 previous similar messages [ 1969.475440] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:26:52 (1785418012) [ 1973.498861] Lustre: 60074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1975.831331] Lustre: 60173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1978.184146] Lustre: 60284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1983.722926] Lustre: 60481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1983.739466] Lustre: 60481:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2018.301255] Lustre: 61951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 2018.317088] Lustre: 61951:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2018.360310] LustreError: 61951:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 2022.732787] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:27:47 (1785418067) [ 2041.830760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.833144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2041.839663] Lustre: Skipped 3 previous similar messages [ 2041.883825] Lustre: Skipped 2 previous similar messages [ 2046.062744] Lustre: server umount lustre-MDT0000 complete [ 2046.946778] LustreError: 51433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.969812] LustreError: 51433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2050.523719] LustreError: 50537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785418097 with bad export cookie 8631075828532050504 [ 2050.527664] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2050.535949] LustreError: 50537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2051.074147] Lustre: server umount lustre-MDT0001 complete [ 2066.860660] Lustre: server umount lustre-OST0000 complete [ 2105.084270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.693444] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2105.765244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2105.772609] Lustre: Skipped 45 previous similar messages [ 2105.978134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2112.458716] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2123.366097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.599432] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2123.682435] Lustre: 64489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2131.769831] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2141.016232] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2148.283314] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2157.527054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2157.822982] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2158.309371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.312506] Lustre: Skipped 1 previous similar message [ 2166.107968] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2174.331474] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2176.114533] Lustre: 67059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2186.721941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2186.741378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.776302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2186.778602] Lustre: Skipped 1 previous similar message [ 2192.360512] LustreError: 66730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.688684] Lustre: server umount lustre-OST0000 complete [ 2201.057146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2201.063460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.072628] Lustre: Skipped 1 previous similar message [ 2207.713987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.735096] Lustre: Skipped 6 previous similar messages [ 2212.320410] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2212.842488] LustreError: 64494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.890306] LustreError: 64494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2212.997979] Lustre: server umount lustre-MDT0000 complete [ 2217.785995] LustreError: 64480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785418264 with bad export cookie 8631075828532053969 [ 2217.799278] LustreError: 64480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2217.799875] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.423654] Lustre: server umount lustre-MDT0001 complete [ 2234.581382] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2238.785535] Key type lgssc unregistered [ 2239.117493] LNet: 68273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.131969] LNetError: 68273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.177418] LNet: Removed LNI 192.168.201.104@tcp [ 2240.415292] Key type .llcrypt unregistered [ 2240.418748] Key type ._llcrypt unregistered [ 2275.556493] Key type ._llcrypt registered [ 2275.559714] Key type .llcrypt registered [ 2275.668875] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2277.304780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2277.340051] alg: No test for adler32 (adler32-zlib) [ 2278.670455] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 2279.105304] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2280.824747] Key type lgssc registered [ 2282.194312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2292.955494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2293.015952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.827112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.559398] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2304.994802] LustreError: 69689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.063745] LustreError: 69689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2310.119523] LustreError: 69688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.491993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.278517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2318.058966] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2328.836723] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2335.971627] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2343.939073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2344.864721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.316732] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2361.801309] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2374.868104] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2388.961416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2388.973596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.991145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.984781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.988049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.048659] Lustre: Skipped 1 previous similar message [ 2391.902770] Lustre: server umount lustre-MDT0000 complete [ 2395.105743] LustreError: 70622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.129661] LustreError: 70622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2396.796890] LustreError: 69669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785418443 with bad export cookie 16778584320832474684 [ 2396.809919] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.300947] Lustre: server umount lustre-MDT0001 complete [ 2412.780655] Lustre: server umount lustre-OST0000 complete [ 2446.710842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.163960] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2447.268711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2447.816948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.096409] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2465.535349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.689887] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2465.752554] Lustre: 74566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2466.524462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2472.655423] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2482.640852] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2490.154837] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2499.177164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2499.684575] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2500.388167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.796940] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2524.896285] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2526.799391] Lustre: 77145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2528.548523] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2539.489218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2539.496839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2539.497616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2539.497620] Lustre: Skipped 2 previous similar messages [ 2539.499629] Lustre: Skipped 1 previous similar message [ 2542.256865] Lustre: server umount lustre-OST0000 complete [ 2548.192671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2548.201342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.227904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.236536] Lustre: Skipped 1 previous similar message [ 2553.080724] Lustre: server umount lustre-MDT0000 complete [ 2554.850285] LustreError: 74578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.880196] LustreError: 74578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2557.739580] LustreError: 74559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785418604 with bad export cookie 16778584320832475531 [ 2557.761100] LustreError: 74559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2557.765734] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.336703] Lustre: server umount lustre-MDT0001 complete [ 2571.819922] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2574.195706] Key type lgssc unregistered [ 2574.603402] LNet: 78354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2574.622091] LNetError: 78354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2575.662179] LNet: Removed LNI 192.168.201.104@tcp [ 2576.333398] Key type .llcrypt unregistered [ 2576.335142] Key type ._llcrypt unregistered [ 2605.500935] Key type ._llcrypt registered [ 2605.504101] Key type .llcrypt registered [ 2605.690891] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2607.708710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2607.861831] alg: No test for adler32 (adler32-zlib) [ 2609.204196] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 2609.483669] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2611.208804] Key type lgssc registered [ 2612.347915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2620.379500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2620.389381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.008367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.687212] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2632.167390] LustreError: 79771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.187554] LustreError: 79771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2637.281304] LustreError: 79772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.402106] LustreError: 79771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.139575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.157155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2652.907906] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2664.088745] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2672.214855] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2682.686578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.834631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.766432] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2702.766770] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2709.591996] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2710.701871] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2712.003515] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2714.271304] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2714.274581] Lustre: Skipped 1 previous similar message [ 2719.780979] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2719.794696] Lustre: Skipped 3 previous similar messages [ 2728.201315] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2728.219649] Lustre: Skipped 6 previous similar messages [ 2745.326912] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2745.330959] Lustre: Skipped 14 previous similar messages [ 2778.189484] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2778.201353] Lustre: Skipped 27 previous similar messages [ 2848.098733] Lustre: 87896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2849.337698] Lustre: Disabling parameter general.jobid_var= in log params [ 2849.342682] Lustre: Skipped 41 previous similar messages [ 2851.307771] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:41:35 (1785418895) [ 2853.147475] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2855.128685] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:41:39 (1785418899) [ 2861.042780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.044495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.049853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2861.057660] Lustre: Skipped 1 previous similar message [ 2861.085878] Lustre: Skipped 1 previous similar message [ 2866.146186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2866.164836] Lustre: Skipped 1 previous similar message [ 2866.680231] Lustre: server umount lustre-OST0000 complete [ 2871.274559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.276685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2871.289125] Lustre: Skipped 1 previous similar message [ 2871.306337] Lustre: Skipped 1 previous similar message [ 2876.384402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.401251] Lustre: Skipped 1 previous similar message [ 2878.148250] Lustre: server umount lustre-MDT0000 complete [ 2881.504512] LustreError: 79772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2881.530164] LustreError: 79772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2882.992872] LustreError: 79752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785418929 with bad export cookie 5135340021192678249 [ 2883.004606] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2883.592939] Lustre: server umount lustre-MDT0001 complete [ 2896.135238] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2899.469971] Key type lgssc unregistered [ 2899.721615] LNet: 89146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2899.728756] LNetError: 89146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2899.748166] LNet: Removed LNI 192.168.201.104@tcp [ 2901.331293] Key type .llcrypt unregistered [ 2901.334063] Key type ._llcrypt unregistered [ 2906.946495] Key type ._llcrypt registered [ 2906.950460] Key type .llcrypt registered [ 2907.138813] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2912.598584] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules [ 2913.981670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2914.176117] alg: No test for adler32 (adler32-zlib) [ 2915.430696] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 2915.811557] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2917.471328] Key type lgssc registered [ 2918.806719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2919.839926] LustreError: 90350:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2941.847324] LustreError: 90350:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2941.937437] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2954.770148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2954.817288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.477778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2962.844971] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2967.009068] LustreError: 91375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.021418] LustreError: 91375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2972.161135] LustreError: 91374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.252451] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:43:41 (1785419021) [ 2977.254180] LustreError: 91375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.380628] LustreError: 91374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.860075] LustreError: 91407:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2985.880807] LustreError: 91407:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2985.897297] LustreError: 91407:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 29, retries 0, failed: rc = -5 [ 2986.677951] Lustre: server umount lustre-MDT0000 complete [ 3007.997101] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 3012.061984] Key type lgssc unregistered [ 3012.436598] LNet: 92903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3012.467303] LNetError: 92903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3012.482267] LNet: Removed LNI 192.168.201.104@tcp [ 3013.669768] Key type .llcrypt unregistered [ 3013.677590] Key type ._llcrypt unregistered [ 3035.299952] Key type ._llcrypt registered [ 3035.301609] Key type .llcrypt registered [ 3035.391567] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3037.169323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3037.261068] alg: No test for adler32 (adler32-zlib) [ 3038.715913] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 3039.214047] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 3040.999194] Key type lgssc registered [ 3043.351876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3053.753594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3053.806442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.925642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.573144] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3065.334323] LustreError: 94128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.353635] LustreError: 94128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3070.443897] LustreError: 94127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.216509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.803677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3080.750936] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3089.340548] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3096.578610] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3106.502737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.960109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.185856] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3126.951743] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3179.741567] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:47:03 (1785419223) [ 3181.800564] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3184.336837] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:47:08 (1785419228) [ 3191.264248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3191.270335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.288481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3192.292578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3192.295443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.321124] Lustre: Skipped 1 previous similar message [ 3196.251513] Lustre: server umount lustre-MDT0000 complete [ 3197.420541] LustreError: 94121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.477727] LustreError: 94121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3200.697039] LustreError: 94105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785419247 with bad export cookie 17560165492971860968 [ 3200.712653] LustreError: 94105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3200.712942] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3201.284235] Lustre: server umount lustre-MDT0001 complete [ 3217.300670] Lustre: server umount lustre-OST0000 complete [ 3231.317625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.871927] LustreError: 98428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.117383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.199610] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3237.361813] LustreError: 98429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.379385] LustreError: 98429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3246.560892] LustreError: 98429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3246.584389] LustreError: 98429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3246.760840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.823792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3254.266082] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3262.244497] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3270.220557] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3285.730809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3286.099184] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3286.116936] LustreError: 98417:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3286.180348] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3286.198199] LustreError: 100515:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3286.212232] LustreError: 100515:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3286.217968] LustreError: 100515:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3286.332555] Lustre: server umount lustre-OST0000 complete [ 3286.335706] LustreError: 100515:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3298.036791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3298.324947] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3298.331295] LustreError: 98417:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3298.355757] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3298.362506] LustreError: 100981:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3298.367721] LustreError: 100981:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3298.372937] LustreError: 100981:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3298.440493] Lustre: server umount lustre-OST0000 complete [ 3298.444513] LustreError: 100981:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3309.726755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3310.026770] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3310.032200] Lustre: Found index 0 for lustre-OST0000, updating log [ 3310.041816] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3310.199286] Lustre: lustre-OST0000: new disk, initializing [ 3310.206077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3310.218758] Lustre: 101517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3310.704444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3318.343701] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3322.521791] Lustre: Failing over lustre-OST0000 [ 3322.996907] Lustre: server umount lustre-OST0000 complete [ 3329.000836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3329.015977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3329.043428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3329.053981] Lustre: Skipped 2 previous similar messages [ 3332.064869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3332.071126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.803880] Lustre: server umount lustre-MDT0000 complete [ 3337.184536] LustreError: 98425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3337.211656] LustreError: 98425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3339.137848] LustreError: 98409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785419385 with bad export cookie 17560165492971863712 [ 3339.142507] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3339.154504] LustreError: 98409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3348.687369] Lustre: server umount lustre-MDT0001 complete [ 3360.279661] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:50:03 (1785419403) [ 3386.404728] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3399.796399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3400.270593] LustreError: 103864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3400.402588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3406.928680] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3418.375242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.382916] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3429.977536] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3439.663427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.460831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.465092] Lustre: Skipped 1 previous similar message [ 3440.476492] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3443.579661] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3443.646539] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3443.652369] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3447.397467] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3458.020481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3458.137901] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3458.237655] Lustre: lustre-OST0001: new disk, initializing [ 3458.241268] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3458.248892] Lustre: 106185:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3465.990435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3466.021180] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3466.299068] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3467.350439] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3476.183285] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3484.811361] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3491.297654] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3491.302828] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.309032] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3491.312132] Lustre: Skipped 1 previous similar message [ 3495.594465] Lustre: server umount lustre-MDT0001 complete [ 3496.762794] LustreError: 103860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3496.793442] LustreError: 103860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3507.726867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.880110] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3507.883586] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3507.915527] Lustre: 103855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3508.738696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3508.761257] Lustre: Skipped 1 previous similar message [ 3509.013178] LustreError: 107710:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3509.028432] LustreError: 107710:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3509.059886] LustreError: 103858:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3509.321829] LustreError: 107725:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.104@tcp: cfg command failed: rc = -17 [ 3509.345441] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3509.345441] [ 3513.835507] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3514.778890] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3530.142447] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:52:54 (1785419574) [ 3553.221414] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3576.332162] Lustre: 109807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3606.553345] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3612.401736] Lustre: 110942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3712.992024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3713.001883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.014265] Lustre: Skipped 2 previous similar messages [ 3713.020620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3713.023891] Lustre: Skipped 3 previous similar messages [ 3715.846910] Lustre: server umount lustre-MDT0000 complete [ 3718.624808] LustreError: 103860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.665874] LustreError: 103860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3723.460697] LustreError: 104608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785419770 with bad export cookie 17560165492971864713 [ 3723.464865] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3723.482772] LustreError: 104608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3724.227228] Lustre: server umount lustre-MDT0001 complete [ 3730.713961] Lustre: server umount lustre-OST0000 complete [ 3736.391069] Lustre: server umount lustre-OST0001 complete [ 3743.370358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3756.138114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3766.802551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3778.838527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3805.247737] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3817.364143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.396664] Lustre: lustre-MDT0000: reset Object Index mappings [ 3817.953257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3823.867485] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3835.022749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.055355] Lustre: lustre-MDT0001: reset Object Index mappings [ 3841.268566] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3844.426836] Lustre: 116063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3852.957586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3861.316946] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3862.411574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3862.777839] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3872.545718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3878.908229] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3878.930314] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3881.206875] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3890.056409] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3893.877839] Lustre: 117938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3955.743679] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:59:59 (1785419999) [ 3981.289633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3981.291479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3981.304619] Lustre: Skipped 8 previous similar messages [ 3981.316484] Lustre: Skipped 9 previous similar messages [ 3987.295504] Lustre: server umount lustre-MDT0000 complete [ 3991.522931] LustreError: 115990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.546064] LustreError: 115990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3991.890116] LustreError: 114896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785420038 with bad export cookie 17560165492971911578 [ 3991.893345] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3991.905422] LustreError: 114896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4006.367254] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4006.788125] Lustre: server umount lustre-MDT0001 complete [ 4017.722268] Lustre: server umount lustre-OST0000 complete [ 4036.063131] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4036.570060] Lustre: server umount lustre-OST0001 complete [ 4043.999505] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 4059.936736] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4110.603838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4110.926803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4110.967934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4111.049950] Lustre: lustre-MDT0000: new disk, initializing [ 4111.177193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.183703] Lustre: Skipped 3 previous similar messages [ 4111.241323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4117.146802] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4125.676976] Lustre: server umount lustre-MDT0000 complete [ 4139.903685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.428412] Lustre: Found index 0 for lustre-MDT0000, updating log [ 4140.459526] Lustre: 122226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4140.485299] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 4146.573256] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4152.752294] Lustre: server umount lustre-MDT0000 complete [ 4164.567763] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:03:28 (1785420208) [ 4166.607075] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 4168.755515] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:03:33 (1785420213) [ 4191.916810] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4203.135513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.951649] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4219.254310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.346795] Lustre: 124275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4219.381610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4219.385025] Lustre: Skipped 1 previous similar message [ 4219.459644] Lustre: lustre-MDT0001: new disk, initializing [ 4219.604127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4219.625808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4224.948456] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4230.802701] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4240.112448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4240.489773] Lustre: lustre-OST0000: new disk, initializing [ 4240.495878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4240.502335] Lustre: 125855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4242.491426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4242.512368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4242.735495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4248.740277] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4263.340537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4263.644684] Lustre: lustre-OST0001: new disk, initializing [ 4263.652654] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4263.663434] Lustre: 126729:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4265.550356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4265.553830] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4265.828226] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4273.515581] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4287.232606] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4292.633467] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4779.546389] Lustre: 133163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4779.556867] Lustre: 133163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4779.565853] Lustre: 133163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4779.575138] Lustre: 133163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4779.583656] Lustre: 133163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4779.591266] Lustre: 133163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 5090.075961] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:18:53 (1785421133) [ 5109.731674] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.733267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5109.744846] Lustre: Skipped 5 previous similar messages [ 5109.760755] Lustre: Skipped 11 previous similar messages [ 5112.179745] Lustre: server umount lustre-MDT0000 complete [ 5114.853440] LustreError: 127751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5114.904770] LustreError: 127751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5116.989713] LustreError: 124267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785421163 with bad export cookie 17560165492971982579 [ 5116.993745] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5116.998202] LustreError: 124267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5119.975709] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.986214] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5119.986654] Lustre: Skipped 1 previous similar message [ 5120.000552] Lustre: Skipped 1 previous similar message [ 5123.888365] Lustre: server umount lustre-MDT0001 complete [ 5134.872750] Lustre: server umount lustre-OST0000 complete [ 5145.492087] Lustre: server umount lustre-OST0001 complete [ 5154.228914] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 5173.696455] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5225.116227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.449942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5225.500768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5225.613854] Lustre: lustre-MDT0000: new disk, initializing [ 5225.755283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5225.766676] Lustre: Skipped 5 previous similar messages [ 5225.781636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5232.362837] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5247.207512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.339841] Lustre: 136245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5247.425923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5247.439894] Lustre: Skipped 1 previous similar message [ 5247.634953] Lustre: lustre-MDT0001: new disk, initializing [ 5247.787757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5247.795225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5252.543960] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5265.919963] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5275.995735] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5285.041138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5285.272979] Lustre: lustre-OST0000: new disk, initializing [ 5285.276978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5285.281913] Lustre: 138216:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5287.383488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5287.394980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5287.495242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5292.933750] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5304.945182] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5318.164640] Lustre: lustre-MDD0000: changelog on [ 5322.282801] Lustre: lustre-MDD0001: changelog on [ 7626.024836] Lustre: lustre-MDD0001: changelog off [ 7628.488560] Lustre: 139676:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7629.214887] Lustre: lustre-MDD0000: changelog off [ 7637.984420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7637.988804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7638.004916] LustreError: Skipped 1 previous similar message [ 7638.006171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7638.043663] Lustre: Skipped 1 previous similar message [ 7638.092052] Lustre: Skipped 3 previous similar messages [ 7643.576527] Lustre: server umount lustre-OST0000 complete [ 7648.745781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7648.773243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7648.776788] Lustre: Skipped 2 previous similar messages [ 7653.860820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7653.867779] Lustre: Skipped 1 previous similar message [ 7654.069676] Lustre: server umount lustre-MDT0000 complete [ 7658.981087] LustreError: 136257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7659.028567] LustreError: 136257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7659.891132] LustreError: 136239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785423706 with bad export cookie 17560165492972766138 [ 7659.907781] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7659.925065] LustreError: 136239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7660.605359] Lustre: server umount lustre-MDT0001 complete [ 7677.064218] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 7680.364380] Key type lgssc unregistered [ 7680.704540] LNet: 147418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7680.710699] LNetError: 147418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7680.731646] LNet: Removed LNI 192.168.201.104@tcp [ 7681.636231] Key type .llcrypt unregistered [ 7681.638351] Key type ._llcrypt unregistered [ 7706.680422] Key type ._llcrypt registered [ 7706.686439] Key type .llcrypt registered [ 7706.877993] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 7727.684769] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7728.990308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7729.113204] alg: No test for adler32 (adler32-zlib) [ 7730.423094] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 7730.755794] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 7732.399165] Key type lgssc registered [ 7733.902450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7786.425188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7786.460746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7787.827753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7787.911661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7788.052950] Lustre: lustre-MDT0000: new disk, initializing [ 7788.207717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.254664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7793.766960] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7806.622949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.754422] Lustre: 150623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7806.810364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7806.813739] Lustre: Skipped 1 previous similar message [ 7806.892742] Lustre: lustre-MDT0001: new disk, initializing [ 7806.979586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7807.012193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7807.027763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7814.112716] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7825.406821] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7832.737350] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7841.683666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7842.093604] Lustre: lustre-OST0000: new disk, initializing [ 7842.097537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7842.104991] Lustre: 152593:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7842.215879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7850.215911] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7851.150289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7851.169510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7851.382945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7862.081033] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7868.604433] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_module obdecho/obdecho [ 7871.344644] LustreError: 153597:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7871.354273] LustreError: 153597:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7874.312460] LustreError: 153694:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7892.453813] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7892.473936] Lustre: Skipped 2 previous similar messages [ 7892.475591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7892.498038] Lustre: Skipped 2 previous similar messages [ 7897.583905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7897.587029] Lustre: Skipped 2 previous similar messages [ 7898.112498] Lustre: server umount lustre-MDT0000 complete [ 7902.655329] LustreError: 150615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785423949 with bad export cookie 17892509019909087801 [ 7902.663463] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7902.678435] LustreError: 150615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7902.724467] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.748555] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7903.375812] Lustre: server umount lustre-MDT0001 complete [ 7908.323077] Lustre: server umount lustre-OST0000 complete [ 7932.472978] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 7944.233113] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7990.872461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7991.038979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7991.053984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7991.108965] Lustre: lustre-MDT0000: new disk, initializing [ 7991.219433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.233847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7999.380363] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8013.258568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.433665] Lustre: 157410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8013.499212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8013.505525] Lustre: Skipped 1 previous similar message [ 8013.611599] Lustre: lustre-MDT0001: new disk, initializing [ 8013.738800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8013.821163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8013.839057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8020.627541] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8031.690570] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8038.884807] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8046.585210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8047.098243] Lustre: lustre-OST0000: new disk, initializing [ 8047.106589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8047.114692] Lustre: 159384:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8047.216767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8054.493491] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8055.373298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8055.384366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8055.475445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8065.493588] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8070.783244] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8070.963280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8076.173928] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8076.528704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8081.381277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.391090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8081.408084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8081.432088] Lustre: Skipped 1 previous similar message [ 8085.526382] Lustre: server umount lustre-OST0000 complete [ 8090.593395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8090.600259] LustreError: Skipped 1 previous similar message [ 8090.602142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8090.615425] Lustre: Skipped 1 previous similar message [ 8090.621093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8096.824938] Lustre: server umount lustre-MDT0000 complete [ 8100.741783] LustreError: 157402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785424147 with bad export cookie 17892509019909088753 [ 8100.745951] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8100.756461] LustreError: 157402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8101.384245] Lustre: server umount lustre-MDT0001 complete [ 8104.986826] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:09:09 (1785424149) [ 8106.467197] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 8108.497583] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:09:12 (1785424152) [ 8118.966992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8119.443292] LustreError: 161798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8119.453381] LustreError: 161798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8119.495240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8124.562760] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8124.896310] LustreError: 161799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8130.037060] LustreError: 161799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8135.137332] LustreError: 161798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8135.542498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8141.252353] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8150.853341] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8157.301941] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8165.499454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8165.858559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8165.862492] Lustre: Skipped 1 previous similar message [ 8173.582672] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8182.396603] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8202.720179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8202.726040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8202.729929] Lustre: Skipped 1 previous similar message [ 8202.731798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8202.734404] Lustre: Skipped 4 previous similar messages [ 8205.203643] Lustre: server umount lustre-MDT0000 complete [ 8208.689037] LustreError: 164095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8209.914255] LustreError: 161780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785424256 with bad export cookie 17892509019909089635 [ 8209.937208] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8215.008319] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8215.027295] Lustre: Skipped 2 previous similar messages [ 8216.853472] Lustre: server umount lustre-MDT0001 complete [ 8224.497257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8239.242246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8239.497707] LustreError: 165666:0:(llog.c:1655:llog_backup()) MGC192.168.201.104@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8239.511334] Lustre: 165666:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.104@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8244.606668] LustreError: 165671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8244.629165] LustreError: 165671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8260.920146] LustreError: 165673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8260.953906] LustreError: 165673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 8295.726679] LustreError: 165671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8295.756164] LustreError: 165671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 8300.511330] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 8300.519453] Lustre: 165666:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 8341.527609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8346.602682] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8347.166273] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8357.424306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8360.247579] LustreError: 165673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8360.264914] LustreError: 165673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 8372.201186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 8387.070454] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8387.156937] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8397.978299] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8399.807788] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8408.481477] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8411.169855] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8438.240380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8438.241468] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8438.258681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8438.275434] Lustre: Skipped 2 previous similar messages [ 8438.292364] Lustre: Skipped 8 previous similar messages [ 8439.209196] Lustre: server umount lustre-MDT0000 complete [ 8443.843710] LustreError: 161778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785424490 with bad export cookie 17892509019909096131 [ 8443.856735] LustreError: 161778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8443.859121] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8444.442946] Lustre: server umount lustre-MDT0001 complete [ 8458.565515] Lustre: server umount lustre-OST0000 complete [ 8486.196634] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 8499.496465] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8542.304655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.577094] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8542.609492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8542.684188] Lustre: lustre-MDT0000: new disk, initializing [ 8542.740900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8542.755784] Lustre: Skipped 1 previous similar message [ 8542.777183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8547.949286] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8560.784252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.993900] Lustre: 170775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8561.384453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8566.147868] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8577.427643] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8585.347738] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8595.030745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8595.250836] Lustre: lustre-OST0000: new disk, initializing [ 8595.253410] Lustre: Skipped 1 previous similar message [ 8595.256964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8595.262848] Lustre: Skipped 2 previous similar messages [ 8595.275221] Lustre: 172745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8602.702845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8602.722876] Lustre: Skipped 1 previous similar message [ 8602.736644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8602.877687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8603.867572] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8615.503279] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8622.871422] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8623.208135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8631.520702] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8631.807833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8638.944481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8638.967798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.006285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8640.951600] Lustre: server umount lustre-OST0000 complete [ 8651.938615] Lustre: server umount lustre-MDT0000 complete [ 8654.305134] LustreError: 170783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8654.325389] LustreError: 170783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 8656.295838] LustreError: 171572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785424703 with bad export cookie 17892509019909112931 [ 8656.297357] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8656.305865] LustreError: 171572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8656.958359] Lustre: server umount lustre-MDT0001 complete [ 8672.122569] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:18:35 (1785424715) [ 8681.503975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.908163] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8698.644652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.830734] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8715.943184] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8723.456144] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8732.291476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8741.039231] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8752.753365] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8767.804582] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:20:11 (1785424811) [ 8777.186933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8777.188972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.213898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8777.213907] Lustre: Skipped 4 previous similar messages [ 8777.214480] LustreError: Skipped 2 previous similar messages [ 8777.259287] Lustre: Skipped 4 previous similar messages [ 8782.141622] Lustre: server umount lustre-OST0000 complete [ 8796.053360] Lustre: server umount lustre-MDT0000 complete [ 8801.578723] LustreError: 175141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785424848 with bad export cookie 17892509019909113813 [ 8801.593366] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8801.594886] LustreError: 175141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8801.910905] Lustre: server umount lustre-MDT0001 complete [ 8817.267675] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 8820.402996] Key type lgssc unregistered [ 8820.842590] LNet: 178898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8820.848268] LNetError: 178898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8820.869926] LNet: Removed LNI 192.168.201.104@tcp [ 8821.979712] Key type .llcrypt unregistered [ 8821.982958] Key type ._llcrypt unregistered [ 8841.435339] Key type ._llcrypt registered [ 8841.437432] Key type .llcrypt registered [ 8841.531908] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8842.734831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8842.963795] alg: No test for adler32 (adler32-zlib) [ 8844.032357] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 8844.268283] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 8846.056926] Key type lgssc registered [ 8847.477366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8856.861476] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8856.894659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8872.543595] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8877.663426] LustreError: 180139:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.104@tcp: failed processing log, type 4: rc = -110 [ 8882.783513] LustreError: 180139:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8882.808299] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8882.823829] LustreError: MGC192.168.201.104@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8882.848345] LustreError: 180139:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8882.857216] LustreError: 180139:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8882.868905] LustreError: 180139:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8882.980673] Lustre: server umount lustre-OST0000 complete [ 8882.991542] LustreError: 180139:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8892.816135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.140364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8899.285121] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8903.653478] LustreError: 180638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8903.681484] LustreError: 180638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8908.771417] LustreError: 180639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8910.510751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8910.988269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8917.190741] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8926.054576] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8934.258535] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8942.285416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8942.559260] LustreError: 182458:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8942.567417] Lustre: 182458:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.104@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8942.785450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8950.505712] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8961.149829] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8967.152734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8967.164680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8967.191058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8968.677437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8968.686685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8968.726500] Lustre: Skipped 2 previous similar messages [ 8970.885383] Lustre: server umount lustre-MDT0000 complete [ 8973.809151] LustreError: 181686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8973.840512] LustreError: 181686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8976.204086] LustreError: 180618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785425022 with bad export cookie 11916972829099532791 [ 8976.207284] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8976.216563] LustreError: 180618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8976.790867] Lustre: server umount lustre-MDT0001 complete [ 8995.213705] Lustre: server umount lustre-OST0000 complete [ 9007.271215] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9020.881524] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9065.394168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9065.851464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9065.912327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9066.071726] Lustre: lustre-MDT0000: new disk, initializing [ 9066.205582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9066.233238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9071.894294] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9085.583720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.838540] Lustre: 185990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9085.921321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9085.925680] Lustre: Skipped 1 previous similar message [ 9086.016691] Lustre: lustre-MDT0001: new disk, initializing [ 9086.136538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9086.187230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9086.208192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9091.544092] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9102.760093] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9110.503394] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9120.706081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9121.151173] Lustre: lustre-OST0000: new disk, initializing [ 9121.160404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9121.171758] Lustre: 187960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9121.293894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9129.144366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9129.160574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9129.343905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9129.774626] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9140.139160] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9145.947949] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9146.253677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9151.947626] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9152.184909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9155.040187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9155.051874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9155.073814] Lustre: Skipped 1 previous similar message [ 9155.082100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9155.101022] Lustre: Skipped 1 previous similar message [ 9160.167131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9160.181586] Lustre: Skipped 1 previous similar message [ 9160.832595] Lustre: server umount lustre-OST0000 complete [ 9168.361583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9168.374199] LustreError: Skipped 1 previous similar message [ 9168.380793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9168.395990] Lustre: Skipped 1 previous similar message [ 9168.409817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9172.694871] Lustre: server umount lustre-MDT0000 complete [ 9175.527982] LustreError: 185998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9175.548439] LustreError: 185998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9177.447242] LustreError: 185982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785425224 with bad export cookie 11916972829099533519 [ 9177.454236] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9177.470056] LustreError: 185982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9177.990300] Lustre: server umount lustre-MDT0001 complete [ 9191.056755] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:27:14 (1785425234) [ 9217.467409] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9221.461603] Key type lgssc unregistered [ 9221.974517] LNet: 190907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9221.980875] LNetError: 190907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9221.995286] LNet: Removed LNI 192.168.201.104@tcp [ 9222.797952] Key type .llcrypt unregistered [ 9222.802764] Key type ._llcrypt unregistered [ 9247.107476] Key type ._llcrypt registered [ 9247.109545] Key type .llcrypt registered [ 9247.282381] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9248.621704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9248.651302] alg: No test for adler32 (adler32-zlib) [ 9250.506808] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 9250.911709] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9252.687230] Key type lgssc registered [ 9254.547804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9264.252273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9264.281632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.819470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9272.573341] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9275.363501] LustreError: 192125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9275.386179] LustreError: 192125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9280.509366] LustreError: 192126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9284.506296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9285.039571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9290.072775] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9302.142093] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9311.732275] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9322.558056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9322.811237] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9322.816771] LustreError: 193991:0:(llog.c:1696:llog_backup()) MGC192.168.201.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 9322.826236] Lustre: 193991:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.104@tcp: can't backup local config lustre-OST0000: rc = -28 [ 9323.027174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9330.331480] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9340.238616] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9346.535652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9346.547195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9346.551622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9346.571353] Lustre: Skipped 2 previous similar messages [ 9351.663216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9351.676103] Lustre: Skipped 2 previous similar messages [ 9356.771263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9356.774090] Lustre: Skipped 2 previous similar messages [ 9357.279741] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9357.857077] Lustre: server umount lustre-MDT0000 complete [ 9361.888625] LustreError: 192120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9361.912193] LustreError: 192120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9362.222969] LustreError: 192106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785425408 with bad export cookie 11874236561048429456 [ 9362.231524] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9362.234432] LustreError: 192106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9362.731485] Lustre: server umount lustre-MDT0001 complete [ 9377.488360] Lustre: server umount lustre-OST0000 complete [ 9388.636695] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9402.843996] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9449.644075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.891718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9449.926469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9450.092390] Lustre: lustre-MDT0000: new disk, initializing [ 9450.250658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.264331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9458.108775] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9473.134576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9473.358957] Lustre: 197519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9473.404986] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9473.406737] Lustre: Skipped 1 previous similar message [ 9473.409616] LustreError: 198307:0:(llog.c:1696:llog_backup()) MGC192.168.201.104@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 9473.413902] LustreError: 198307:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 9473.418129] Lustre: 198307:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.104@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 9473.432243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9473.442833] Lustre: Skipped 1 previous similar message [ 9473.524360] Lustre: lustre-MDT0001: new disk, initializing [ 9473.618205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9473.646541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9473.664368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9479.616400] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9489.558965] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9497.070134] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9505.141257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9505.429914] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9505.432061] Lustre: Skipped 1 previous similar message [ 9505.438607] LustreError: 199468:0:(llog.c:1696:llog_backup()) MGC192.168.201.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 9505.443610] LustreError: 199468:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 9505.447598] Lustre: 199468:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.104@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 9505.452929] Lustre: 199468:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 9505.479394] Lustre: lustre-OST0000: new disk, initializing [ 9505.482088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9505.486536] Lustre: 199491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9505.553516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9512.927301] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9513.587136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9513.595346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9513.796371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9524.946200] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9532.238751] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9532.563600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9537.809099] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9537.965703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9544.676597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9544.681232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9544.687927] LustreError: Skipped 1 previous similar message [ 9544.688957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9544.738533] Lustre: Skipped 3 previous similar messages [ 9549.797330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9549.807579] Lustre: Skipped 2 previous similar messages [ 9554.399445] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9554.706592] Lustre: server umount lustre-OST0000 complete [ 9565.152880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9565.159635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9565.206114] Lustre: Skipped 1 previous similar message [ 9565.249401] Lustre: Skipped 1 previous similar message [ 9568.441230] Lustre: server umount lustre-MDT0000 complete [ 9570.278699] LustreError: 198649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9570.309459] LustreError: 198649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9573.465337] LustreError: 197514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785425620 with bad export cookie 11874236561048430177 [ 9573.472971] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9573.477238] LustreError: 197514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9573.823454] Lustre: server umount lustre-MDT0001 complete [ 9584.612945] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:33:49 (1785425629) [ 9607.507108] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9618.854521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9619.198380] LustreError: 202635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9619.256957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9624.329514] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9628.737641] LustreError: 202635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9628.771269] LustreError: 202635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9634.603930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9641.678109] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9644.907796] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9653.164422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9653.832408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9653.842253] Lustre: Skipped 1 previous similar message [ 9662.752479] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9673.570991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9673.784142] Lustre: lustre-OST0001: new disk, initializing [ 9673.791881] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9673.798418] Lustre: 204946:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9682.138213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9682.149681] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9682.272332] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9684.733514] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9701.080470] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9711.657975] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9715.001227] Lustre: DEBUG MARKER: ADD OST3 [ 9721.915533] loop: module loaded [ 9721.991547] loop0: detected capacity change from 0 to 204800000 [ 9722.348500] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9722.395650] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9722.420495] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9730.661750] Lustre: DEBUG MARKER: STOP OST3 [ 9731.153415] loop0: detected capacity change from 0 to 204800000 [ 9738.993881] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9739.244489] Lustre: lustre-OST0003: new disk, initializing [ 9739.254277] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 9739.263817] Lustre: 207002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9739.467727] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 9739.472320] Lustre: Skipped 1 previous similar message [ 9741.172740] Lustre: *** cfs_fail_loc=2109, val=2*** [ 9741.282210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 9741.296840] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 9741.330840] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 9742.730019] Lustre: Failing over lustre-OST0003 [ 9743.188042] Lustre: server umount lustre-OST0003 complete [ 9743.199115] LustreError: 207138:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 9746.269871] LustreError: 204124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9746.289202] LustreError: 204124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9746.400460] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9746.411200] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.490582] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9756.971310] Lustre: DEBUG MARKER: START OST3 again [ 9764.187941] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.685085] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9765.802449] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9766.690777] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9766.692484] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9772.925735] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9797.942206] Lustre: Failing over lustre-OST0003 [ 9798.276333] Lustre: server umount lustre-OST0003 complete [ 9800.694342] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9800.718404] Lustre: Skipped 2 previous similar messages [ 9800.733515] LustreError: 205244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9800.752563] LustreError: 205244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 9819.616799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9819.631824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9819.653536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9824.174841] Lustre: server umount lustre-MDT0000 complete [ 9827.916937] LustreError: 202615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785425874 with bad export cookie 11874236561048431059 [ 9827.920298] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9827.930860] LustreError: 202615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9834.581166] Lustre: server umount lustre-MDT0001 complete [ 9839.226698] Lustre: server umount lustre-OST0000 complete [ 9843.818395] Lustre: server umount lustre-OST0001 complete [ 9852.492501] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9864.964662] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9918.461284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9918.960445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9919.070031] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9919.459634] Lustre: lustre-MDT0000: new disk, initializing [ 9919.761733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.769619] Lustre: Skipped 1 previous similar message [ 9919.815891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9926.874334] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9943.906552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9944.284965] Lustre: 212415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9944.440506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9944.445823] Lustre: Skipped 1 previous similar message [ 9944.767087] Lustre: lustre-MDT0001: new disk, initializing [ 9945.156175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9945.173602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9954.717518] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9968.934732] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9977.942183] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9988.221514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9988.892677] Lustre: lustre-OST0000: new disk, initializing [ 9988.910524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9988.945441] Lustre: 214388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9990.720762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9990.738907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9990.938913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9999.618637] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10011.995673] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10021.520040] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10021.962659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10027.649407] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10027.953172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10032.111331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10032.121202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10032.127514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10032.127523] Lustre: Skipped 8 previous similar messages [10032.132472] Lustre: Skipped 6 previous similar messages [10032.145220] LustreError: Skipped 1 previous similar message [10044.383155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10044.812848] Lustre: server umount lustre-OST0000 complete [10052.579722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10057.293965] Lustre: server umount lustre-MDT0000 complete [10057.699033] LustreError: 212423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10057.728857] LustreError: 212423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [10062.276548] LustreError: 213213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785426109 with bad export cookie 11874236561048433348 [10062.284488] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10062.285690] LustreError: 213213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10062.688339] Lustre: server umount lustre-MDT0001 complete [10067.546945] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:41:51 (1785426111) [10090.719804] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [10105.319897] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10152.158259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10152.512961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10152.540950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10152.605728] Lustre: lustre-MDT0000: new disk, initializing [10152.692723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10158.475319] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10175.363815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10175.556434] Lustre: 219007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10175.915287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10175.921249] Lustre: Skipped 3 previous similar messages [10175.992615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10182.688830] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10198.122707] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10206.449269] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10216.683673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10217.146859] Lustre: 220981:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10218.894201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10219.070095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10225.112856] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10236.540940] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10243.189896] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10243.552841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10249.140741] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10249.421793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10254.816056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10254.817664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10254.835041] LustreError: Skipped 1 previous similar message [10254.843134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10254.843142] Lustre: Skipped 7 previous similar messages [10254.868889] Lustre: Skipped 3 previous similar messages [10258.365992] Lustre: server umount lustre-OST0000 complete [10270.104564] Lustre: server umount lustre-MDT0000 complete [10270.182623] LustreError: 219019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10270.194664] LustreError: 219019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10275.200987] LustreError: 218999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785426321 with bad export cookie 11874236561048434230 [10275.205462] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10275.216620] LustreError: 218999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10276.120290] Lustre: server umount lustre-MDT0001 complete [10287.096496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10294.046163] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10304.736865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10312.054980] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10320.835845] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10327.351342] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10336.350358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10345.006452] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10353.711038] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10372.070465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10372.084274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10372.097462] Lustre: Skipped 2 previous similar messages [10373.633166] Lustre: server umount lustre-OST0000 complete [10383.344065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10383.350298] Lustre: Skipped 8 previous similar messages [10385.256646] Lustre: server umount lustre-MDT0000 complete [10389.910532] LustreError: 223178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785426436 with bad export cookie 11874236561048435112 [10389.937989] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10390.461636] Lustre: server umount lustre-MDT0001 complete [10498.654406] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10502.068751] Key type lgssc unregistered [10502.428781] LNet: 226886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10502.436067] LNetError: 226886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10502.448853] LNet: Removed LNI 192.168.201.104@tcp [10503.413290] Key type .llcrypt unregistered [10503.424037] Key type ._llcrypt unregistered [10540.807492] Key type ._llcrypt registered [10540.809117] Key type .llcrypt registered [10540.980493] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [10562.167666] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10564.346577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10564.361910] alg: No test for adler32 (adler32-zlib) [10565.635545] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [10566.041825] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [10567.887647] Key type lgssc registered [10569.651181] Lustre: Echo OBD driver; http://www.lustre.org/ [10628.699051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10628.735326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10630.070043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10630.110987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10630.236691] Lustre: lustre-MDT0000: new disk, initializing [10630.399718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10630.426924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10636.559237] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10648.651689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.841512] Lustre: 230244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10648.929200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10648.940515] Lustre: Skipped 1 previous similar message [10649.093162] Lustre: lustre-MDT0001: new disk, initializing [10649.211885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10649.313588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10649.323552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10655.585490] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10669.454296] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10677.323225] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10688.593267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10689.050548] Lustre: lustre-OST0000: new disk, initializing [10689.056915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10689.066936] Lustre: 232214:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10689.203778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10697.643093] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10698.908704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10698.919264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10699.050537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10709.776035] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10717.267285] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10717.553538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10726.313815] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10726.574468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10729.953223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10729.969857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10729.974151] LustreError: Skipped 1 previous similar message [10729.975372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10730.013146] Lustre: Skipped 1 previous similar message [10735.078711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10735.111840] Lustre: Skipped 2 previous similar messages [10740.231854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10740.242307] Lustre: Skipped 1 previous similar message [10743.263340] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10743.965278] Lustre: server umount lustre-OST0000 complete [10751.456810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10751.469611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10751.489166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10755.563545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10755.566574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10755.621117] Lustre: Skipped 1 previous similar message [10765.279402] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10765.929090] Lustre: server umount lustre-MDT0000 complete [10770.819850] LustreError: 232224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785426817 with bad export cookie 4260011460888166038 [10770.831350] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10770.884298] LustreError: 232224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10771.437845] Lustre: server umount lustre-MDT0001 complete [10803.220664] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10817.727161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.183579] LustreError: 235188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.203772] LustreError: 235188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10818.265025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10823.653826] LustreError: 235189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10824.224498] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10827.748314] LustreError: 235188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10832.881712] LustreError: 235189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10836.131975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10836.877148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10842.556397] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10848.363000] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10859.993055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10860.475535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10869.014875] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10880.342038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10880.522800] Lustre: lustre-OST0001: new disk, initializing [10880.526755] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10880.532617] Lustre: 237501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10880.620939] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10887.333090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10887.347700] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10887.518968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10887.939086] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10901.278687] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10911.014431] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10926.780457] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:56:10 (1785426970) [10929.122572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10929.131165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10929.151979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10929.154704] Lustre: Skipped 2 previous similar messages [10933.732600] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10933.750361] Lustre: Skipped 2 previous similar messages [10935.386786] Lustre: server umount lustre-MDT0000 complete [10938.854066] LustreError: 235185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.887436] LustreError: 235185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10940.891186] LustreError: 235169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785426987 with bad export cookie 4260011460888166920 [10940.902453] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10940.914978] LustreError: 235169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10941.588686] Lustre: server umount lustre-MDT0001 complete [10956.079528] Lustre: server umount lustre-OST0000 complete [10959.839585] Lustre: 228467:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785426990/real 1785426990] req@ffffa0ebd08e8700 x1872155495231360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785427006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10959.872750] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10960.732877] Lustre: server umount lustre-OST0001 complete [10970.616034] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [10984.473690] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11030.856210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11031.282829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11031.382976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11031.582954] Lustre: lustre-MDT0000: new disk, initializing [11031.785069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11031.832045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11038.668721] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11054.747505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11054.940355] Lustre: 241463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11055.027682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11055.030688] Lustre: Skipped 1 previous similar message [11055.146683] Lustre: lustre-MDT0001: new disk, initializing [11055.350807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11055.366758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11061.386388] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11073.814735] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11082.683596] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11095.241864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11095.701416] Lustre: lustre-OST0000: new disk, initializing [11095.704886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11095.716841] Lustre: 243436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11095.839428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11095.853508] Lustre: Skipped 1 previous similar message [11096.928045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11096.952438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11097.323258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11108.533756] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11120.472511] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11125.900446] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11126.128773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11132.343320] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11132.589625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11138.016279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11138.021303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11138.043359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11138.046879] Lustre: Skipped 4 previous similar messages [11141.510796] Lustre: server umount lustre-OST0000 complete [11147.746948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11152.159934] Lustre: server umount lustre-MDT0000 complete [11153.890846] LustreError: 241472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11153.917912] LustreError: 241472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11157.385893] LustreError: 241455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785427204 with bad export cookie 4260011460888169048 [11157.388851] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11157.399565] LustreError: 241455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11158.486652] Lustre: server umount lustre-MDT0001 complete [11169.910618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11170.387319] LustreError: 245654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.468354] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11187.086930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11195.407984] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11205.424761] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11213.455205] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11222.558434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11232.279724] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11242.111775] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11389.431515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11389.436184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11389.453210] Lustre: Skipped 3 previous similar messages [11389.455945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11389.455951] Lustre: Skipped 4 previous similar messages [11403.231387] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11403.658105] Lustre: server umount lustre-OST0000 complete [11415.018496] LustreError: 245655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11415.029861] LustreError: 245655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11415.072630] Lustre: server umount lustre-MDT0000 complete [11420.032520] LustreError: 245634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785427466 with bad export cookie 4260011460888169937 [11420.037890] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11420.040149] LustreError: 245634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11420.551861] Lustre: server umount lustre-MDT0001 complete [11514.401795] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [11520.369216] Key type lgssc unregistered [11520.962443] LNet: 249276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11520.978375] LNetError: 249276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11521.029485] LNet: Removed LNI 192.168.201.104@tcp [11523.124132] Key type .llcrypt unregistered [11523.128192] Key type ._llcrypt unregistered [11565.152312] Key type ._llcrypt registered [11565.155095] Key type .llcrypt registered [11565.328789] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [11586.988289] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11588.794572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11589.008360] alg: No test for adler32 (adler32-zlib) [11590.600138] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [11591.194250] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [11593.111941] Key type lgssc registered [11595.865962] Lustre: Echo OBD driver; http://www.lustre.org/ [11664.562853] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11680.269326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11680.359917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11681.709044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11681.765417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11681.950591] Lustre: lustre-MDT0000: new disk, initializing [11682.115361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11682.179792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11689.642522] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11705.367674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11705.462350] Lustre: 253389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11705.566697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11705.573959] Lustre: Skipped 1 previous similar message [11705.726694] Lustre: lustre-MDT0001: new disk, initializing [11705.907657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11705.961056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11705.973841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11712.182089] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11718.382503] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11727.746069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11728.189163] Lustre: lustre-OST0000: new disk, initializing [11728.192137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11728.200777] Lustre: 255022:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11728.321973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11729.818884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11729.838869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11730.045774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11736.793238] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11753.198872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11753.579438] Lustre: lustre-OST0001: new disk, initializing [11753.586680] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11753.594894] Lustre: 255897:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11753.850874] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11759.757815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11759.778760] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11759.867133] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11761.660515] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11774.929138] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11785.430854] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11803.105434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11803.122806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.149263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11806.185718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11806.188225] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11806.202973] Lustre: Skipped 1 previous similar message [11806.243233] Lustre: Skipped 1 previous similar message [11809.557691] Lustre: server umount lustre-MDT0000 complete [11816.223993] LustreError: 255895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785427862 with bad export cookie 14877572522055654344 [11816.240331] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11816.248853] LustreError: 255895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11816.418723] LustreError: 253395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11816.423101] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.447297] LustreError: 253395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [11816.475399] Lustre: Skipped 1 previous similar message [11816.486373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11816.500195] Lustre: Skipped 3 previous similar messages [11817.188100] Lustre: server umount lustre-MDT0001 complete [11824.610836] Lustre: server umount lustre-OST0000 complete [11830.926691] Lustre: server umount lustre-OST0001 complete [11849.430807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11863.478597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11874.573627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11887.004169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11911.698783] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11925.105600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11925.126427] Lustre: lustre-MDT0000: reset Object Index mappings [11925.431730] LustreError: 260915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11925.511363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11930.606206] LustreError: 260916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.240548] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11935.722611] LustreError: 260915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11940.856881] LustreError: 260916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11944.464616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11944.475925] Lustre: lustre-MDT0001: reset Object Index mappings [11944.886233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11951.296571] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11955.615212] Lustre: 262057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11964.668779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11965.295625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11969.433502] LustreError: 262412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11970.515867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [11975.251210] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11985.892731] LustreError: 262585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11985.940617] LustreError: 262585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11988.003426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11993.616145] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [12000.050053] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12011.694453] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12016.156354] Lustre: 263931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12024.292220] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12024.296105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12024.315924] Lustre: Skipped 3 previous similar messages [12024.347633] Lustre: Skipped 3 previous similar messages [12029.325028] Lustre: server umount lustre-MDT0000 complete [12029.420809] LustreError: 260915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12029.444579] LustreError: 260915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12033.506078] LustreError: 260897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785428080 with bad export cookie 14877572522055661134 [12033.506882] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12033.519837] LustreError: 260897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12034.039292] Lustre: server umount lustre-MDT0001 complete [12048.843833] Lustre: server umount lustre-OST0000 complete [12050.596801] Lustre: 250865:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785428081/real 1785428081] req@ffffa0ebc1a51c00 x1872156570040960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785428097 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12050.628808] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12050.637493] Lustre: Skipped 1 previous similar message [12054.350564] Lustre: server umount lustre-OST0001 complete [12078.632315] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:15:23 (1785428123) [12098.188740] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [12110.358099] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12162.893867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12163.359943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12163.409489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12163.553951] Lustre: lustre-MDT0000: new disk, initializing [12163.726388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12163.735573] Lustre: Skipped 1 previous similar message [12163.780872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12170.159597] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12183.379614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12183.546902] Lustre: 268191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12183.642724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12183.650899] Lustre: Skipped 1 previous similar message [12183.819562] Lustre: lustre-MDT0001: new disk, initializing [12184.045538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12184.055546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12190.677298] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12205.788365] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12214.351368] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12225.758976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12226.219706] Lustre: lustre-OST0000: new disk, initializing [12226.229963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12226.253087] Lustre: 270165:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12228.018558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12228.041898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12228.344233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12235.783905] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12251.668497] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12258.616705] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12259.108440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12265.363232] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12265.583729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12269.044530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12269.058367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12269.096175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12274.667074] LustreError: 270151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12274.897774] Lustre: server umount lustre-OST0000 complete [12281.312274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12281.327243] LustreError: Skipped 1 previous similar message [12281.348631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12281.356458] Lustre: Skipped 1 previous similar message [12286.414780] Lustre: server umount lustre-MDT0000 complete [12291.036170] LustreError: 268181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785428337 with bad export cookie 14877572522055663227 [12291.060985] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12291.698273] Lustre: server umount lustre-MDT0001 complete [12319.299511] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12332.627735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12333.350857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12333.361745] Lustre: Skipped 2 previous similar messages [12342.432931] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12356.886980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12364.022224] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12366.975459] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12376.408000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12385.589759] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12396.997618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12397.313828] Lustre: lustre-OST0001: new disk, initializing [12397.331336] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12397.350408] Lustre: 275429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12399.182932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12399.208920] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12399.305685] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12406.994552] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12419.344169] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12424.396437] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12429.979886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [12430.081730] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [12439.522919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12439.525856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12439.546959] Lustre: Skipped 3 previous similar messages [12439.553175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12439.563796] Lustre: Skipped 2 previous similar messages [12445.673375] LustreError: 273122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12445.711328] LustreError: 273122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [12445.801244] Lustre: server umount lustre-MDT0000 complete [12452.419320] LustreError: 275440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785428499 with bad export cookie 14877572522055664109 [12452.447455] LustreError: 275440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12452.453402] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12453.859306] Lustre: server umount lustre-MDT0001 complete [12472.287147] Lustre: 250863:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785428502/real 1785428502] req@ffffa0ebd28d6d80 x1872156570168192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785428518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12472.338970] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12472.350623] Lustre: Skipped 3 previous similar messages [12473.311205] Lustre: 250865:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785428504/real 1785428504] req@ffffa0ebd28d4a80 x1872156570168448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785428520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12473.447730] Lustre: server umount lustre-OST0000 complete [12476.449283] Lustre: 250863:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785428507/real 1785428507] req@ffffa0ed01448700 x1872156570168704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785428523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12479.895618] Lustre: server umount lustre-OST0001 complete [12504.970107] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12517.868550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12518.439936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12518.451569] Lustre: Skipped 3 previous similar messages [12524.359413] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12536.881971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12545.744322] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12550.024238] Lustre: 279634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12559.164266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12563.990194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [12568.894307] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12580.578326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12585.989735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [12589.158892] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12600.217542] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12606.377496] Lustre: 281511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12614.193183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [12614.429185] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [12627.950917] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:24:32 (1785428672) [12647.394578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12647.396522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12647.411961] Lustre: Skipped 2 previous similar messages [12647.436482] Lustre: Skipped 6 previous similar messages [12653.568091] Lustre: server umount lustre-MDT0000 complete [12658.821355] LustreError: 278473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785428705 with bad export cookie 14877572522055670913 [12658.831490] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12658.842674] LustreError: 278473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12665.730113] Lustre: server umount lustre-MDT0001 complete [12677.457491] Lustre: server umount lustre-OST0000 complete [12681.868552] Lustre: server umount lustre-OST0001 complete [12691.792923] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [12710.296417] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12773.133566] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12785.993221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12786.223154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12786.269946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12786.405849] Lustre: lustre-MDT0000: new disk, initializing [12786.553047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12786.562149] Lustre: Skipped 3 previous similar messages [12786.585695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12792.517562] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12807.076688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12807.263182] Lustre: 285776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12807.668985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12813.339912] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12819.424054] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12828.121313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12828.355582] Lustre: 287412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12829.754463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12836.694415] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12850.810790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12851.056641] Lustre: 288283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12852.584260] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12852.604749] Lustre: Skipped 1 previous similar message [12852.819699] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12859.936793] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12873.284310] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12899.516614] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:29:03 (1785428943) [12903.915081] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12903.925807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12903.935233] Lustre: Skipped 4 previous similar messages [12903.946958] Lustre: Skipped 7 previous similar messages [12909.855722] Lustre: server umount lustre-MDT0000 complete [12914.159126] LustreError: 286626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12914.184625] LustreError: 286626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 32 previous similar messages [12914.308892] LustreError: 285770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785428961 with bad export cookie 14877572522055678368 [12914.313797] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12914.325729] LustreError: 285770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12915.002766] Lustre: server umount lustre-MDT0001 complete [12930.063468] Lustre: server umount lustre-OST0000 complete [12944.362238] Lustre: server umount lustre-OST0001 complete [12956.007363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12963.057287] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12974.502705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12980.490141] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12990.356734] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12997.836427] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13007.424624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13017.264594] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13028.242583] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13058.141407] loop1: detected capacity change from 0 to 204800000 [13058.497395] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13058.527201] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13058.556295] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13064.281458] loop1: detected capacity change from 0 to 204800000 [13071.323090] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13071.432311] LustreError: 290524:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs4886: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [13071.444532] LustreError: 290524:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs4886 [13071.449222] LustreError: 290524:0:(mgs_handler.c:517:mgs_target_reg()) fs4886-MDT0001: Error checking target index: rc = -13 [13071.457416] LustreError: fs4886-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [13071.465920] LustreError: 294259:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [13071.474062] LustreError: 294259:0:(tgt_mount.c:2129:server_put_super()) no obd fs4886-MDT0001 [13071.479427] LustreError: 294259:0:(tgt_mount.c:116:server_deregister_mount()) fs4886-MDT0001 not registered [13071.510317] Lustre: server umount fs4886-MDT0001 complete [13071.514859] LustreError: 294259:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [13083.724042] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13083.874435] Lustre: Setting parameter fs4886-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs4886-MDT0001 [13083.884498] Lustre: Skipped 1 previous similar message [13083.997699] Lustre: srv-fs4886-MDT0001: No data found on store. Initialize space. [13084.007319] Lustre: Skipped 4 previous similar messages [13084.171703] Lustre: fs4886-MDT0001: new disk, initializing [13084.179935] Lustre: Skipped 3 previous similar messages [13084.365504] Lustre: srv-fs4886-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13084.381080] Lustre: 294727:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4886-MDT0001-mdtlov: can't create new llog: rc = -115 [13084.402318] LustreError: 294727:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4886-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [13092.512558] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13095.943360] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [13101.039480] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [13102.829291] Lustre: server umount fs4886-MDT0001 complete [13121.387866] loop1: detected capacity change from 0 to 204800000 [13129.501722] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13130.317865] Lustre: 296297:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4886-MDT0001-mdtlov: can't create new llog: rc = -115 [13130.332711] LustreError: 296297:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4886-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [13137.695758] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13144.823680] Lustre: server umount fs4886-MDT0001 complete [13159.396541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13159.412409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13159.418697] Lustre: Skipped 4 previous similar messages [13162.469156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13162.485535] Lustre: Skipped 4 previous similar messages [13165.159559] Lustre: server umount lustre-MDT0000 complete [13175.285888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13175.343553] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13180.944110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13182.059509] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13195.510761] loop1: detected capacity change from 0 to 204800000 [13206.273084] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13207.039848] Lustre: 298747:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4886-MDT0001-mdtlov: can't create new llog: rc = -115 [13207.051314] LustreError: 298747:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4886-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [13213.829238] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13219.900559] Lustre: server umount fs4886-MDT0001 complete [13247.491734] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [13252.614077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13265.378845] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13265.603093] Lustre: server umount lustre-OST0000 complete [13273.064893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13277.542796] Lustre: server umount lustre-MDT0000 complete [13282.356488] LustreError: 290518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785429329 with bad export cookie 14877572522055681049 [13282.358469] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13282.369406] LustreError: 290518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13282.990615] Lustre: server umount lustre-MDT0001 complete [13301.024547] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13304.194313] Key type lgssc unregistered [13304.744859] LNet: 301118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13304.759261] LNetError: 301118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13304.776338] LNet: Removed LNI 192.168.201.104@tcp [13305.595196] Key type .llcrypt unregistered [13305.596966] Key type ._llcrypt unregistered [13329.971763] Key type ._llcrypt registered [13329.975812] Key type .llcrypt registered [13330.124496] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [13331.515723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13331.579233] alg: No test for adler32 (adler32-zlib) [13332.915421] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [13333.274528] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [13334.999221] Key type lgssc registered [13336.371830] Lustre: Echo OBD driver; http://www.lustre.org/ [13348.400311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13348.447151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13350.142544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13356.443433] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13359.598246] LustreError: 302390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13359.627137] LustreError: 302390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13364.707356] LustreError: 302391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13367.930613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13368.433824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13376.195198] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13388.950252] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13397.681368] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13409.036528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13409.601079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13413.673443] LustreError: 304222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13418.621395] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13419.005021] LustreError: 304224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13425.121580] LustreError: 304225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13425.150631] LustreError: 304225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13429.423860] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13435.367157] LustreError: 304222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13435.418702] LustreError: 304222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13452.123624] LustreError: 304224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13452.155849] LustreError: 304224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [13453.493438] loop1: detected capacity change from 0 to 204800000 [13453.794345] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13453.831426] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13453.858221] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13459.979483] loop1: detected capacity change from 0 to 204800000 [13468.151795] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13468.355443] Lustre: fs15350-OST0000: new disk, initializing [13468.357527] Lustre: srv-fs15350-OST0000: No data found on store. Initialize space. [13468.363134] Lustre: 306126:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13468.453807] Lustre: fs15350-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13475.846387] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13485.899177] Lustre: server umount fs15350-OST0000 complete [13486.563274] LustreError: 304224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13486.601428] LustreError: 304224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [13491.702815] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [13496.805815] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [13501.217799] loop1: detected capacity change from 0 to 204800000 [13508.122362] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13508.473150] Lustre: fs15350-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13516.135952] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13527.880033] loop2: detected capacity change from 0 to 204800000 [13528.208886] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13528.227225] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13528.256939] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13534.728477] loop2: detected capacity change from 0 to 204800000 [13542.616454] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13542.762650] LustreError: fs15350-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [13542.780471] LustreError: fs15350-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [13542.787812] LustreError: 309166:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [13542.795893] LustreError: 309166:0:(tgt_mount.c:2129:server_put_super()) no obd fs15350-OST0001 [13542.801359] LustreError: 309166:0:(tgt_mount.c:116:server_deregister_mount()) fs15350-OST0001 not registered [13542.847206] Lustre: server umount fs15350-OST0001 complete [13542.850468] LustreError: 309166:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [13553.125157] LustreError: 304222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13553.168647] LustreError: 304222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [13556.941840] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13557.242759] Lustre: fs15350-OST0001: new disk, initializing [13557.247850] Lustre: srv-fs15350-OST0001: No data found on store. Initialize space. [13557.266742] Lustre: 309718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13557.382436] Lustre: fs15350-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13567.977087] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13580.530382] Lustre: server umount fs15350-OST0000 complete [13592.697112] Lustre: server umount fs15350-OST0001 complete [13609.634755] loop1: detected capacity change from 0 to 204800000 [13616.920366] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13617.410656] Lustre: fs15350-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13624.691462] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13633.201638] loop2: detected capacity change from 0 to 204800000 [13640.796307] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13645.287427] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [13649.531092] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13650.420173] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [13662.418947] Lustre: server umount fs15350-OST0000 complete [13675.860482] Lustre: server umount fs15350-OST0001 complete [13681.137108] LustreError: 304225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13681.185052] LustreError: 304225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 74 previous similar messages [13696.484204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13696.496768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13696.525766] Lustre: Skipped 1 previous similar message [13696.584365] Lustre: Skipped 1 previous similar message [13701.604429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13701.619438] Lustre: Skipped 1 previous similar message [13702.848983] Lustre: server umount lustre-OST0000 complete [13711.329562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13711.342335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13711.366962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13717.132481] Lustre: server umount lustre-MDT0000 complete [13723.106796] LustreError: 302372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785429769 with bad export cookie 2260346581008412546 [13723.107830] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13723.125614] LustreError: 302372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13724.028354] Lustre: server umount lustre-MDT0001 complete [13740.358590] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13743.710385] Key type lgssc unregistered [13744.112553] LNet: 315651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13744.132778] LNetError: 315651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13744.146856] LNet: Removed LNI 192.168.201.104@tcp [13745.076135] Key type .llcrypt unregistered [13745.078497] Key type ._llcrypt unregistered [13790.160950] Key type ._llcrypt registered [13790.179225] Key type .llcrypt registered [13790.451143] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [13811.471810] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [13812.624334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13812.823148] alg: No test for adler32 (adler32-zlib) [13814.172444] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [13814.394993] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [13816.056931] Key type lgssc registered [13817.760226] Lustre: Echo OBD driver; http://www.lustre.org/ [13862.064678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13862.099640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13863.445172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13863.491866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13863.602722] Lustre: lustre-MDT0000: new disk, initializing [13863.674137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13863.686815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13868.088218] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13880.883935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13881.052723] Lustre: 319517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13881.127414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13881.132059] Lustre: Skipped 1 previous similar message [13881.267980] Lustre: lustre-MDT0001: new disk, initializing [13881.395320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13881.456683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13881.473879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13887.581235] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13901.085622] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13910.380635] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13920.475219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13920.898436] Lustre: lustre-OST0000: new disk, initializing [13920.903194] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13920.913687] Lustre: 321486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13921.020782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13926.513619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13926.530167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13926.665404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13929.693255] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [13942.312103] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13945.295494] Lustre: Mounted lustre-client [13946.900541] Lustre: Unmounted lustre-client [13960.685918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13960.712251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13960.718567] Lustre: Skipped 1 previous similar message [13965.613106] Lustre: server umount lustre-OST0000 complete [13973.474080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13973.477683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13973.488499] Lustre: Skipped 1 previous similar message [13973.494661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13976.033373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13976.041692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13977.503641] Lustre: server umount lustre-MDT0000 complete [13982.535681] LustreError: 319508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785430029 with bad export cookie 10779628815862091141 [13982.546370] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13982.550776] LustreError: 319508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13983.176174] Lustre: server umount lustre-MDT0001 complete [13997.792667] LNet: 1 local NIs in recovery (showing 1): 192.168.201.104@tcp [13999.569423] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [14004.792964] Key type lgssc unregistered [14005.681069] LNet: 323651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14005.704430] LNetError: 323651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14075.374848] LNet: Removed LNI 192.168.201.104@tcp [14076.806311] Key type .llcrypt unregistered [14076.809394] Key type ._llcrypt unregistered [14101.636420] Key type ._llcrypt registered [14101.637915] Key type .llcrypt registered [14101.734208] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [14127.066383] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [14128.542026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14128.646222] alg: No test for adler32 (adler32-zlib) [14130.243693] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [14130.766600] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [14132.599405] Key type lgssc registered [14134.126174] Lustre: Echo OBD driver; http://www.lustre.org/ [14173.692383] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:50:17 (1785430217) [14182.894925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14182.958457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14184.333844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14184.382287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14184.513157] Lustre: lustre-MDT0000: new disk, initializing [14184.618691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14184.657318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14190.714772] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14203.916551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14204.039845] Lustre: 326879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14204.090570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14204.094230] Lustre: Skipped 1 previous similar message [14204.157311] Lustre: lustre-MDT0001: new disk, initializing [14204.264630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14204.300203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14204.318313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14209.732357] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14222.509033] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14229.636767] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14238.831940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14239.155153] Lustre: lustre-OST0000: new disk, initializing [14239.157854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14239.188953] Lustre: 328848:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14239.358587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14247.601653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14247.621063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14247.864857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14248.263504] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14262.919746] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14288.878160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14288.885533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14288.906269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14288.908959] LustreError: Skipped 1 previous similar message [14288.948754] Lustre: Skipped 1 previous similar message [14290.484424] Lustre: server umount lustre-OST0000 complete [14296.544920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14296.554855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14296.569289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14296.575159] Lustre: Skipped 1 previous similar message [14299.105720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14299.107447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14301.796392] Lustre: server umount lustre-MDT0000 complete [14308.649308] LustreError: 326871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785430355 with bad export cookie 13420116657771820214 [14308.659995] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14308.680945] LustreError: 326871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14310.025541] Lustre: server umount lustre-MDT0001 complete [14328.205547] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [14332.144697] Key type lgssc unregistered [14332.562039] LNet: 330759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14332.570529] LNetError: 330759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14332.591938] LNet: Removed LNI 192.168.201.104@tcp [14333.874849] Key type .llcrypt unregistered [14333.876394] Key type ._llcrypt unregistered [14365.447290] Key type ._llcrypt registered [14365.448839] Key type .llcrypt registered [14365.582217] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [14367.077120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14367.151049] alg: No test for adler32 (adler32-zlib) [14369.227450] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [14369.778614] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [14371.535720] Key type lgssc registered [14373.829333] Lustre: Echo OBD driver; http://www.lustre.org/ [14383.910606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14383.931833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14385.273570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14392.058670] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14394.850328] LustreError: 332032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14394.870633] LustreError: 332032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14399.977588] LustreError: 332033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14405.040281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14405.090688] LustreError: 332032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14405.369963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14413.299344] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14425.605767] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14432.459819] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14443.000570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14443.711699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14452.393593] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14463.716808] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14476.286378] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14478.117637] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14485.994136] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14486.002789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14486.021518] Lustre: Skipped 2 previous similar messages [14486.051773] Lustre: Skipped 2 previous similar messages [14489.739062] Lustre: server umount lustre-MDT0000 complete [14491.107343] LustreError: 332889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14491.138306] LustreError: 332889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14494.324631] LustreError: 332014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785430541 with bad export cookie 1928918811647059527 [14494.335276] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14494.340485] LustreError: 332014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14496.249592] LustreError: 332027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14496.257291] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14496.270848] LustreError: 332027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14496.303874] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14496.311523] Lustre: Skipped 1 previous similar message [14499.232857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14499.243542] Lustre: Skipped 1 previous similar message [14500.779430] Lustre: server umount lustre-MDT0001 complete [14504.965212] Lustre: server umount lustre-OST0000 complete [14515.157673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14522.094670] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14525.222459] Lustre: 336220:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.104@tcp [14527.563297] Lustre: server umount MGS complete [14540.212527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14540.620590] LustreError: 336793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14540.644699] LustreError: 336793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14540.789056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14547.128376] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14557.154637] LustreError: 336794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14557.172664] LustreError: 336794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14560.141165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14560.632957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14566.865629] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14577.696659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14578.004478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14587.430997] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14600.415167] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14602.524975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [14611.785334] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14613.869677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14622.178598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14622.186901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14622.200341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14623.224937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14626.863919] Lustre: server umount lustre-OST0000 complete [14632.416580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14632.426250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14632.441569] Lustre: Skipped 1 previous similar message [14632.447904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14632.453793] Lustre: Skipped 2 previous similar messages [14638.560516] LustreError: 336788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14638.575212] LustreError: 336788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14638.861804] Lustre: server umount lustre-MDT0000 complete [14643.466807] LustreError: 336773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785430690 with bad export cookie 1928918811647060367 [14643.480496] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14643.482096] LustreError: 336773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14644.064393] Lustre: server umount lustre-MDT0001 complete [14662.094414] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [14665.654944] Key type lgssc unregistered [14665.922257] LNet: 340232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14665.926969] LNetError: 340232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14665.938850] LNet: Removed LNI 192.168.201.104@tcp [14666.573925] Key type .llcrypt unregistered [14666.579234] Key type ._llcrypt unregistered [14693.876743] Key type ._llcrypt registered [14693.883674] Key type .llcrypt registered [14694.050951] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [14716.353522] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [14718.767375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14718.866047] alg: No test for adler32 (adler32-zlib) [14720.235676] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [14720.581462] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [14722.311127] Key type lgssc registered [14723.925536] Lustre: Echo OBD driver; http://www.lustre.org/ [14769.190848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14769.266353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14770.738329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14770.772379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14770.941566] Lustre: lustre-MDT0000: new disk, initializing [14771.078903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14771.108457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14778.032394] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14791.140374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14791.419952] Lustre: 343400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14791.549460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14791.559862] Lustre: Skipped 1 previous similar message [14791.758459] Lustre: lustre-MDT0001: new disk, initializing [14791.994494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14792.143272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14792.166901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14800.194592] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14812.302504] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14820.460592] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14830.939255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14831.765196] Lustre: lustre-OST0000: new disk, initializing [14831.790453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14831.818705] Lustre: 345385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14832.092979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14837.954157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14837.978622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14838.236437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14844.050844] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [14859.427858] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14866.220747] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14866.529525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14873.112902] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14873.373141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14879.203295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14879.210466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14879.225503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14879.245680] Lustre: Skipped 1 previous similar message [14879.271643] Lustre: Skipped 1 previous similar message [14882.070277] Lustre: server umount lustre-OST0000 complete [14889.443824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14889.449788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14889.459917] LustreError: Skipped 1 previous similar message [14889.463797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14889.514139] Lustre: Skipped 1 previous similar message [14893.266300] Lustre: server umount lustre-MDT0000 complete [14898.656343] LustreError: 343393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785430945 with bad export cookie 5518311146802420290 [14898.665127] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14898.671639] LustreError: 343393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14899.848497] Lustre: server umount lustre-MDT0001 complete [14913.931396] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:02:37 (1785430957) [14938.327519] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [14950.891931] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [15000.624960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15001.044316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15001.081030] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15001.258265] Lustre: lustre-MDT0000: new disk, initializing [15001.467831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15001.499595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15010.208528] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [15025.279985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15025.362333] Lustre: 350152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15025.407289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15025.409544] Lustre: Skipped 1 previous similar message [15025.501505] Lustre: lustre-MDT0001: new disk, initializing [15025.656348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15025.696389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15025.701821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15030.830364] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [15043.185983] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15052.012440] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15061.233486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15061.623961] Lustre: lustre-OST0000: new disk, initializing [15061.627400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15061.637736] Lustre: 352123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15061.734937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15070.060653] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [15070.866544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15070.887853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15071.034302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15083.831936] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15137.770963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15137.772076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15137.777533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15137.793974] Lustre: Skipped 1 previous similar message [15137.836098] Lustre: Skipped 2 previous similar messages [15139.320597] Lustre: server umount lustre-OST0000 complete [15148.002634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15148.035728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15148.047286] Lustre: Skipped 1 previous similar message [15151.834514] Lustre: server umount lustre-MDT0000 complete [15152.037054] LustreError: 350163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15152.070407] LustreError: 350163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15157.217041] LustreError: 350162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15157.294078] LustreError: 350162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15158.249313] LustreError: 350159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15158.275302] LustreError: 350159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15158.684661] LustreError: 352122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785431205 with bad export cookie 5518311146802421172 [15158.695323] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15158.705979] LustreError: 352122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15159.524445] Lustre: server umount lustre-MDT0001 complete [15178.687747] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15182.955179] Key type lgssc unregistered [15183.329036] LNet: 354323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15183.336079] LNetError: 354323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15183.363860] LNet: Removed LNI 192.168.201.104@tcp [15184.574263] Key type .llcrypt unregistered [15184.580557] Key type ._llcrypt unregistered [15308.036606] Key type ._llcrypt registered [15308.038467] Key type .llcrypt registered [15308.183727] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15308.554287] Key type .llcrypt unregistered [15308.556908] Key type ._llcrypt unregistered [15418.613402] Key type ._llcrypt registered [15418.614774] Key type .llcrypt registered [15418.891380] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15419.469355] Key type .llcrypt unregistered [15419.477551] Key type ._llcrypt unregistered [15457.384830] Key type ._llcrypt registered [15457.387150] Key type .llcrypt registered [15457.586693] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15458.027987] Key type .llcrypt unregistered [15458.031220] Key type ._llcrypt unregistered [15504.012112] Key type ._llcrypt registered [15504.018246] Key type .llcrypt registered [15504.313193] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15504.596415] Key type .llcrypt unregistered [15504.606137] Key type ._llcrypt unregistered [15604.946878] Key type ._llcrypt registered [15604.950262] Key type .llcrypt registered [15605.049536] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15605.360477] Key type .llcrypt unregistered [15605.366569] Key type ._llcrypt unregistered [15647.225962] Key type ._llcrypt registered [15647.228224] Key type .llcrypt registered [15647.381686] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15647.646550] Key type .llcrypt unregistered [15647.651533] Key type ._llcrypt unregistered [15686.718166] Key type ._llcrypt registered [15686.719722] Key type .llcrypt registered [15686.870147] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15687.143286] Key type .llcrypt unregistered [15687.144861] Key type ._llcrypt unregistered [15724.689586] Key type ._llcrypt registered [15724.691050] Key type .llcrypt registered [15724.826077] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15725.224617] Key type .llcrypt unregistered [15725.229419] Key type ._llcrypt unregistered [15777.322959] Key type ._llcrypt registered [15777.327020] Key type .llcrypt registered [15777.481050] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15777.971310] Key type .llcrypt unregistered [15777.978379] Key type ._llcrypt unregistered [15815.673207] Key type ._llcrypt registered [15815.680871] Key type .llcrypt registered [15815.912307] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15816.494380] Key type .llcrypt unregistered [15816.496094] Key type ._llcrypt unregistered [15897.619233] Key type ._llcrypt registered [15897.622904] Key type .llcrypt registered [15897.767779] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15898.150643] Key type .llcrypt unregistered [15898.152696] Key type ._llcrypt unregistered [15938.051464] Key type ._llcrypt registered [15938.055948] Key type .llcrypt registered [15938.378939] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15939.006385] Key type .llcrypt unregistered [15939.008159] Key type ._llcrypt unregistered [15976.969171] Key type ._llcrypt registered [15976.974031] Key type .llcrypt registered [15977.172647] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [15977.599471] Key type .llcrypt unregistered [15977.605551] Key type ._llcrypt unregistered [16051.490882] Key type ._llcrypt registered [16051.495844] Key type .llcrypt registered [16051.730718] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [16074.581328] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [16076.384360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16076.557182] alg: No test for adler32 (adler32-zlib) [16077.701464] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [16078.189113] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [16079.951266] Key type lgssc registered [16081.767357] Lustre: Echo OBD driver; http://www.lustre.org/ [16130.321751] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16130.359902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16131.687916] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [16131.725920] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [16131.852669] Lustre: tst250fs-MDT0000: new disk, initializing [16132.009767] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16132.031718] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16137.806690] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [16150.114401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16150.267398] Lustre: 374741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16150.334650] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [16150.341553] Lustre: Skipped 1 previous similar message [16150.439548] Lustre: tst250fs-MDT0001: new disk, initializing [16150.561054] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16150.596784] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16150.608184] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16156.358315] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [16161.929990] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [16170.685139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16171.095912] Lustre: tst250fs-OST0000: new disk, initializing [16171.100988] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [16171.122867] Lustre: 376373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16171.268969] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16176.769169] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16176.785131] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16176.916183] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16178.767774] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [16194.248174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16194.684989] Lustre: tst250fs-OST0001: new disk, initializing [16194.690592] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [16194.703804] Lustre: 377245:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16194.847772] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [16202.439700] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [16202.454553] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [16202.562706] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [16204.868844] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [16214.572085] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [16218.275665] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [16218.288626] Lustre: Skipped 1 previous similar message [16242.671381] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16242.677994] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16242.735670] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [16243.689225] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [16243.707528] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16243.747242] Lustre: Skipped 2 previous similar messages [16249.051328] Lustre: server umount tst250fs-MDT0000 complete [16253.921457] LustreError: 374753:0:(ldlm_lib.c:1192:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16253.950700] LustreError: 374753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [16254.123564] LustreError: 374731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785432300 with bad export cookie 12346866263150197126 [16254.133525] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16254.447634] Lustre: server umount tst250fs-MDT0001 complete [16270.207249] Lustre: server umount tst250fs-OST0000 complete [16285.054625] Lustre: server umount tst250fs-OST0001 complete [16308.628855] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [16318.778236] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [16360.380496] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:26:44 (1785432404) [16362.511189] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [16386.004128] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [16408.394580] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [16454.689848] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16148 sec ========= 13:28:18 (1785432498) [16457.498922] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:28:21 (1785432501) === [16462.521936] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:28:26 (1785432506) === [16520.644453] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [16525.160679] Key type lgssc unregistered [16525.566352] LNet: 386407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16525.580027] LNetError: 386407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16525.611836] LNet: Removed LNI 192.168.201.104@tcp [16527.975219] Key type .llcrypt unregistered [16527.978378] Key type ._llcrypt unregistered