[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 449970019 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003074] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005027] kvm-guest: setup PV IPIs [ 0.008575] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.012054] LSM: Security Framework initializing [ 0.013048] Yama: becoming mindful. [ 0.014038] SELinux: Initializing. [ 0.015067] *** VALIDATE selinux *** [ 0.024446] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029186] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031058] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033114] *** VALIDATE tmpfs *** [ 0.035261] *** VALIDATE proc *** [ 0.037113] *** VALIDATE cgroup *** [ 0.038009] *** VALIDATE cgroup2 *** [ 0.039269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041043] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043001] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044029] Spectre V2 : User space: Vulnerable [ 0.045011] Speculative Store Bypass: Vulnerable [ 0.048045] debug: unmapping init [mem 0xffffffffb0859000-0xffffffffb0860fff] [ 0.050176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051833] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052018] ... version: 2 [ 0.053010] ... bit width: 48 [ 0.054010] ... generic registers: 4 [ 0.055012] ... value mask: 0000ffffffffffff [ 0.056011] ... max period: 00007fffffffffff [ 0.057012] ... fixed-purpose events: 3 [ 0.058011] ... event mask: 000000070000000f [ 0.059291] rcu: Hierarchical SRCU implementation. [ 0.061474] smp: Bringing up secondary CPUs ... [ 0.062550] x86: Booting SMP configuration: [ 0.063021] .... node #0, CPUs: #1 #2 #3 [ 0.066306] smp: Brought up 1 node, 4 CPUs [ 0.068012] smpboot: Max logical packages: 1 [ 0.069017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145000] node 0 deferred pages initialised in 73ms [ 0.148445] devtmpfs: initialized [ 0.149206] x86/mm: Memory block size: 128MB [ 0.152952] gcov: version magic: 0x41383552 [ 0.156054] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162310] pinctrl core: initialized pinctrl subsystem [ 0.164129] [ 0.164604] ************************************************************* [ 0.167011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169009] ** ** [ 0.171011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173012] ** ** [ 0.176012] ** This means that this kernel is built to expose internal ** [ 0.179012] ** IOMMU data structures, which may compromise security on ** [ 0.181014] ** your system. ** [ 0.184012] ** ** [ 0.186011] ** If you see this message and you are not debugging the ** [ 0.188071] ** kernel, report this immediately to your vendor! ** [ 0.191013] ** ** [ 0.193011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195012] ************************************************************* [ 0.197772] NET: Registered protocol family 16 [ 0.200451] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.205061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208476] cpuidle: using governor menu [ 0.210060] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211434] PCI: Using configuration type 1 for base access [ 0.212206] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220090] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.226121] cryptd: max_cpu_qlen set to 1000 [ 0.229247] ACPI: Added _OSI(Module Device) [ 0.231012] ACPI: Added _OSI(Processor Device) [ 0.232014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239176] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.245284] ACPI: Interpreter enabled [ 0.247058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248011] ACPI: Using IOAPIC for interrupt routing [ 0.250111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253368] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263455] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.271071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276274] acpiphp: Slot [2] registered [ 0.277146] acpiphp: Slot [5] registered [ 0.279113] acpiphp: Slot [6] registered [ 0.280128] acpiphp: Slot [7] registered [ 0.282116] acpiphp: Slot [8] registered [ 0.284080] acpiphp: Slot [9] registered [ 0.285118] acpiphp: Slot [10] registered [ 0.287110] acpiphp: Slot [3] registered [ 0.288093] acpiphp: Slot [4] registered [ 0.290109] acpiphp: Slot [11] registered [ 0.291084] acpiphp: Slot [12] registered [ 0.292090] acpiphp: Slot [13] registered [ 0.294101] acpiphp: Slot [14] registered [ 0.295147] acpiphp: Slot [15] registered [ 0.296081] acpiphp: Slot [16] registered [ 0.298087] acpiphp: Slot [17] registered [ 0.299079] acpiphp: Slot [18] registered [ 0.300087] acpiphp: Slot [19] registered [ 0.302082] acpiphp: Slot [20] registered [ 0.303097] acpiphp: Slot [21] registered [ 0.304089] acpiphp: Slot [22] registered [ 0.306086] acpiphp: Slot [23] registered [ 0.307083] acpiphp: Slot [24] registered [ 0.308091] acpiphp: Slot [25] registered [ 0.309086] acpiphp: Slot [26] registered [ 0.311141] acpiphp: Slot [27] registered [ 0.312108] acpiphp: Slot [28] registered [ 0.314099] acpiphp: Slot [29] registered [ 0.315093] acpiphp: Slot [30] registered [ 0.317087] acpiphp: Slot [31] registered [ 0.318049] PCI host bridge to bus 0000:00 [ 0.319017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.333025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.335183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.339025] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342355] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.351534] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.357012] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.360017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.362015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.365018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.368386] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370822] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.376885] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.394992] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.401018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408910] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.417015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.461000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.470013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.495017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.504067] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.512015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.518017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.535015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.545535] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.554013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.564015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.588013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.602846] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.612015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.619016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.636062] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.650179] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.660014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.667015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.689017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.703366] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.706362] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.708358] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.711348] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.714228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.718150] iommu: Default domain type: Passthrough [ 0.720431] SCSI subsystem initialized [ 0.722153] ACPI: bus type USB registered [ 0.724097] usbcore: registered new interface driver usbfs [ 0.725167] usbcore: registered new interface driver hub [ 0.727099] usbcore: registered new device driver usb [ 0.729152] pps_core: LinuxPPS API ver. 1 registered [ 0.732012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.734054] PTP clock support registered [ 0.737070] EDAC MC: Ver: 3.0.0 [ 0.739151] PCI: Using ACPI for IRQ routing [ 0.740799] NetLabel: Initializing [ 0.741008] NetLabel: domain hash size = 128 [ 0.742008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.743067] NetLabel: unlabeled traffic allowed by default [ 0.744101] vgaarb: loaded [ 0.745255] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.747013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.754303] clocksource: Switched to clocksource kvm-clock [ 0.851703] VFS: Disk quotas dquot_6.6.0 [ 0.853326] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.855975] *** VALIDATE ramfs *** [ 0.857258] *** VALIDATE hugetlbfs *** [ 0.858804] pnp: PnP ACPI init [ 0.861233] pnp: PnP ACPI: found 6 devices [ 0.879515] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.883194] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.885421] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.887699] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.890808] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.892957] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.895581] NET: Registered protocol family 2 [ 0.898375] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.902726] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.906291] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.911639] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.914950] TCP: Hash tables configured (established 65536 bind 65536) [ 0.917728] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.920686] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923440] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.926331] NET: Registered protocol family 1 [ 0.929166] RPC: Registered named UNIX socket transport module. [ 0.931449] RPC: Registered udp transport module. [ 0.933456] RPC: Registered tcp transport module. [ 0.935341] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.938387] NET: Registered protocol family 44 [ 0.940380] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.943041] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.945588] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.948299] PCI: CLS 0 bytes, default 64 [ 0.951153] Unpacking initramfs... [ 2.343095] debug: unmapping init [mem 0xffff979e7cc54000-0xffff979e7ffbffff] [ 2.346020] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.348408] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.351915] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.841222] Initialise system trusted keyrings [ 2.843153] Key type blacklist registered [ 2.845373] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.854887] zbud: loaded [ 2.857835] *** VALIDATE nfs *** [ 2.859009] *** VALIDATE nfs4 *** [ 2.861136] pstore: using deflate compression [ 2.867256] Platform Keyring initialized [ 2.979888] NET: Registered protocol family 38 [ 2.981808] Key type asymmetric registered [ 2.983587] Asymmetric key parser 'x509' registered [ 2.985514] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.989116] io scheduler mq-deadline registered [ 2.992133] io scheduler kyber registered [ 2.993934] io scheduler bfq registered [ 2.996118] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.999369] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.003655] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.006647] ACPI: Power Button [PWRF] [ 3.012570] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.018649] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.030107] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.039580] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.058525] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.085823] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.115411] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.120214] Non-volatile memory driver v1.3 [ 3.121685] Linux agpgart interface v0.103 [ 3.149248] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.151547] vda: detected capacity change from 0 to 75083776 [ 3.164502] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.167727] vdb: detected capacity change from 0 to 1073741824 [ 3.183281] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.186187] vdc: detected capacity change from 0 to 2621440000 [ 3.202023] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.205281] vdd: detected capacity change from 0 to 2621440000 [ 3.219153] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.222559] vde: detected capacity change from 0 to 4294967296 [ 3.237498] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.240561] vdf: detected capacity change from 0 to 4294967296 [ 3.246282] libphy: Fixed MDIO Bus: probed [ 3.251521] usbcore: registered new interface driver usbserial_generic [ 3.253394] usbserial: USB Serial support registered for generic [ 3.255327] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.258596] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.259941] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.261805] mousedev: PS/2 mouse device common for all mice [ 3.265439] rtc_cmos 00:05: RTC can wake from S4 [ 3.268777] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.268846] rtc_cmos 00:05: registered as rtc0 [ 3.274129] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.276983] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.278246] intel_pstate: CPU model not supported [ 3.284760] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.290772] hid: raw HID events driver (C) Jiri Kosina [ 3.293210] usbcore: registered new interface driver usbhid [ 3.295302] usbhid: USB HID core driver [ 3.297191] drop_monitor: Initializing network drop monitor service [ 3.299902] Initializing XFRM netlink socket [ 3.302262] NET: Registered protocol family 10 [ 3.305157] Segment Routing with IPv6 [ 3.306821] NET: Registered protocol family 17 [ 3.308996] mpls_gso: MPLS GSO support [ 3.314286] RAS: Correctable Errors collector initialized. [ 3.316618] AVX version of gcm_enc/dec engaged. [ 3.318330] AES CTR mode by8 optimization enabled [ 3.382502] sched_clock: Marking stable (3382473830, 0)->(4295149303, -912675473) [ 3.385422] registered taskstats version 1 [ 3.388311] Loading compiled-in X.509 certificates [ 3.391546] zswap: loaded using pool lzo/zbud [ 3.415014] Key type big_key registered [ 3.428833] Key type encrypted registered [ 3.430532] ima: No TPM chip found, activating TPM-bypass! [ 3.432818] ima: Allocated hash algorithm: sha1 [ 3.434744] ima: No architecture policies found [ 3.436705] evm: Initialising EVM extended attributes: [ 3.438738] evm: security.selinux [ 3.440124] evm: security.ima [ 3.442157] evm: security.capability [ 3.443537] evm: HMAC attrs: 0x1 [ 3.446130] rtc_cmos 00:05: setting system clock to 2026-09-07 10:01:19 UTC (1788775279) [ 3.453442] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.456804] debug: unmapping init [mem 0xffffffffb0582000-0xffffffffb0858fff] [ 3.466079] Write protecting the kernel read-only data: 28672k [ 3.469686] debug: unmapping init [mem 0xffffffffaec03000-0xffffffffaedfffff] [ 3.472019] debug: unmapping init [mem 0xffffffffaf514000-0xffffffffaf5fffff] [ 3.502987] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.510419] systemd[1]: Detected virtualization kvm. [ 3.511980] systemd[1]: Detected architecture x86-64. [ 3.513529] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.536923] systemd[1]: No hostname configured. [ 3.538456] systemd[1]: Set hostname to . [ 3.540223] random: systemd: uninitialized urandom read (16 bytes read) [ 3.542194] systemd[1]: Initializing machine ID from random generator. [ 3.575481] random: ln: uninitialized urandom read (6 bytes read) [ 3.670202] random: systemd: uninitialized urandom read (16 bytes read) [ 3.672532] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.676342] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.680335] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.212601] device-mapper: uevent: version 1.0.3 [ 4.215314] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 4.762451] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.941607] virtio_net virtio0 ens2: renamed from eth0 [ 4.965339] scsi host0: ata_piix [ 4.978387] scsi host1: ata_piix [ 4.980301] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.983831] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.477516] dracut-initqueue[594]: RTNETLINK answers: File exists [ 9.948528] random: crng init done [ 9.949880] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.364736] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.666484] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.980349] SELinux: Disabled at runtime. [ 12.048199] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.058632] systemd[1]: Detected virtualization kvm. [ 12.060957] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.556803] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.560315] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.565459] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.569936] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.574131] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.582369] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.588350] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ 12.798418] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.032936] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.315095] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.328075] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.432360] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.444376] EDAC sbridge: Ver: 1.1.2 [ 14.854116] Key type dns_resolver registered [ 15.151822] NFS: Registering the id_resolver key type [ 15.153961] Key type id_resolver registered [ 15.157037] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg351-server login: [ 41.077416] libcfs: loading out-of-tree module taints kernel. [ 41.097531] Key type ._llcrypt registered [ 41.098967] Key type .llcrypt registered [ 41.152440] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 56.550486] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 58.197067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.219646] alg: No test for adler32 (adler32-zlib) [ 59.909479] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 61.542851] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 63.432162] Key type lgssc registered [ 65.035322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.719947] hrtimer: interrupt took 6959533 ns [ 83.653848] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 131.899159] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 146.261200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.310891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.605603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.642848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.733930] Lustre: lustre-MDT0000: new disk, initializing [ 147.792490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.813262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.741113] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.895106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.029645] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.081580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.085741] Lustre: Skipped 1 previous similar message [ 167.199621] Lustre: lustre-MDT0001: new disk, initializing [ 167.321446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.362604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 167.380754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 171.920561] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.714878] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.058074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.348947] Lustre: lustre-OST0000: new disk, initializing [ 186.355691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.365507] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.435153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.253590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 188.264951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 188.363332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.773804] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.613733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.706658] Lustre: lustre-OST0001: new disk, initializing [ 207.710821] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.717133] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.770296] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.738990] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.089178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 216.096184] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 216.134238] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 225.682853] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 233.081752] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 239.191074] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing check_logdir /tmp/testlogs/ [ 244.345980] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing yml_node [ 250.895752] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 254.341766] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 258.269098] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 261.404030] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 06:05:35 EDT 2026 [ 281.837889] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 283.490450] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 300.512747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 300.519351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.536761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.075258] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.076873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.090229] Lustre: Skipped 2 previous similar messages [ 303.096727] Lustre: Skipped 3 previous similar messages [ 306.022681] Lustre: server umount lustre-MDT0000 complete [ 309.086167] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775585 with bad export cookie 1822740350687147043 [ 309.096626] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.098290] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 313.313156] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.317283] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.318949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.339567] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 313.369926] Lustre: Skipped 1 previous similar message [ 314.336680] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.347724] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 315.481451] Lustre: server umount lustre-MDT0001 complete [ 318.765645] Lustre: server umount lustre-OST0000 complete [ 322.542815] Lustre: server umount lustre-OST0001 complete [ 330.623450] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 342.121519] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 384.746733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.066868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 385.163572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 385.412711] Lustre: lustre-MDT0000: new disk, initializing [ 385.588696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 385.626156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 393.167811] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 407.361565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.458423] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 407.510338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 407.514545] Lustre: Skipped 1 previous similar message [ 407.598252] Lustre: lustre-MDT0001: new disk, initializing [ 407.724623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 407.783995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 407.793347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 412.318290] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 422.272716] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 428.934293] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 437.173901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 437.513690] Lustre: lustre-OST0000: new disk, initializing [ 437.522705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 437.544697] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 437.641561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.211741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 439.223257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 439.328094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 444.496581] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 453.774518] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 458.588696] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 458.855461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 463.639590] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 463.893777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 469.989209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 469.992208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.004470] LustreError: Skipped 1 previous similar message [ 470.021774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 470.021780] Lustre: Skipped 3 previous similar messages [ 475.107050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.119419] Lustre: Skipped 2 previous similar messages [ 480.224228] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 480.372585] Lustre: server umount lustre-OST0000 complete [ 484.834952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 484.844285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.861309] Lustre: Skipped 1 previous similar message [ 484.865420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 484.872421] Lustre: Skipped 2 previous similar messages [ 499.168217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 499.431580] Lustre: server umount lustre-MDT0000 complete [ 500.713160] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.731710] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 503.828622] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775779 with bad export cookie 1822740350687148576 [ 503.830029] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 503.850945] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 504.365363] Lustre: server umount lustre-MDT0001 complete [ 510.761657] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:09:45 (1788775785) [ 527.927568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.278930] LustreError: 19927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.359597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.460434] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 533.476215] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.478123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.551583] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 554.541035] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 560.155388] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 562.478901] Lustre: Failing over lustre-MDT0000 [ 562.657361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 562.675871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.695339] Lustre: Skipped 1 previous similar message [ 562.701259] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.710273] Lustre: server umount lustre-MDT0000 complete [ 562.719921] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 579.040298] Lustre: 3643:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788775839/real 1788775839] req@ffff979dc8010e00 x1875666887933952/t0(0) o400->MGC192.168.203.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788775855 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 579.080723] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.105190] LustreError: 19927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.133163] LustreError: 19927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 582.446398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.287481] LustreError: 3641:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff979ef772d880 x1875666887936768/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 593.890875] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 594.921359] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 594.926575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 594.964613] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 603.687481] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 609.251294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.260763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 609.270418] Lustre: Skipped 2 previous similar messages [ 609.276587] Lustre: Skipped 6 previous similar messages [ 622.048177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 622.291853] Lustre: server umount lustre-MDT0000 complete [ 625.669550] LustreError: 19908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775901 with bad export cookie 1822740350687149990 [ 625.670328] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.688477] LustreError: 19908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 626.114900] Lustre: server umount lustre-MDT0001 complete [ 635.210480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.546352] LustreError: 23265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.568757] LustreError: 23265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 635.628802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.636163] Lustre: Skipped 2 previous similar messages [ 640.194970] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 650.066304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.212780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 650.222104] Lustre: Skipped 7 previous similar messages [ 654.756737] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 662.362521] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 667.868458] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 670.066563] Lustre: Failing over lustre-MDT0000 [ 670.295338] Lustre: server umount lustre-MDT0000 complete [ 670.692864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 670.701819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.585803] Lustre: 3645:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788775947/real 1788775947] req@ffff979ec97baa00 x1875666887962752/t0(0) o400->MGC192.168.203.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788775963 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 687.619080] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 690.083485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.825193] LustreError: 3641:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff979dc5ce4000 x1875666887965568/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 698.160314] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 702.392797] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 703.461861] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 703.468274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 703.497384] Lustre: Skipped 1 previous similar message [ 703.519379] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 713.230919] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 717.741460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 732.128519] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 732.378410] Lustre: server umount lustre-MDT0000 complete [ 736.532406] LustreError: 23248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776012 with bad export cookie 1822740350687150928 [ 736.534439] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 736.556598] LustreError: 23248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 736.949208] Lustre: server umount lustre-MDT0001 complete [ 746.446623] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:13:41 (1788776021) [ 762.422597] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 773.592837] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 817.357086] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 827.007880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.294184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 827.355725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 827.459621] Lustre: lustre-MDT0000: new disk, initializing [ 827.538113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.547780] Lustre: Skipped 2 previous similar messages [ 827.579232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 832.049398] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 842.856312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.989957] Lustre: 29641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 843.208890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 848.065375] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 857.998688] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 864.863444] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 871.824546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.070942] Lustre: 31609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 873.987991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 874.047736] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 874.162607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 874.176842] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 878.856427] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 888.325123] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 894.114317] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 915.425577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 915.441042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.445440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 915.470212] Lustre: Skipped 4 previous similar messages [ 915.480462] Lustre: Skipped 8 previous similar messages [ 921.485835] Lustre: server umount lustre-OST0000 complete [ 931.752399] Lustre: server umount lustre-MDT0000 complete [ 935.737807] LustreError: 32834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776211 with bad export cookie 1822740350687151327 [ 935.741354] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 935.753377] LustreError: 32834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 936.273143] Lustre: server umount lustre-MDT0001 complete [ 948.044278] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 951.244021] Key type lgssc unregistered [ 951.553215] LNet: 33584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.570813] LNetError: 33584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.595559] LNet: Removed LNI 192.168.203.151@tcp [ 952.727185] Key type .llcrypt unregistered [ 952.729913] Key type ._llcrypt unregistered [ 976.235114] Key type ._llcrypt registered [ 976.237506] Key type .llcrypt registered [ 976.359677] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 977.512954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 977.535063] alg: No test for adler32 (adler32-zlib) [ 978.603503] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 978.846665] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 980.528187] Key type lgssc registered [ 981.569670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 989.834832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 989.851431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 991.285144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.969688] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1001.452096] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.466746] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1005.044762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.498160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1010.281489] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1018.553402] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1024.344356] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1031.765531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1032.143458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.511815] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1048.427099] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1050.551922] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1065.034306] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:18:59 (1788776339) [ 1067.414875] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1069.913667] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1077.738377] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:19:12 (1788776352) [ 1086.995819] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:19:22 (1788776362) [ 1088.111706] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1096.757206] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1096.761615] Lustre: Skipped 2 previous similar messages [ 1098.488277] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:19:33 (1788776373) [ 1102.230662] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1102.233797] Lustre: Skipped 3 previous similar messages [ 1121.087884] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1121.093379] Lustre: Skipped 6 previous similar messages [ 1124.078447] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:19:59 (1788776399) [ 1126.885799] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1130.776177] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1139.335780] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:20:14 (1788776414) [ 1141.281292] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1141.286123] Lustre: Skipped 3 previous similar messages [ 1154.090896] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:20:29 (1788776429) [ 1169.380521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1169.400428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.420183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.486870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1173.488553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.496578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.520049] Lustre: Skipped 1 previous similar message [ 1174.729467] Lustre: server umount lustre-OST0000 complete [ 1179.623172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1179.630235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.650207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.720174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.722118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1185.917531] Lustre: server umount lustre-MDT0000 complete [ 1188.840257] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.869733] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1190.175213] LustreError: 34978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776466 with bad export cookie 4413251035602238573 [ 1190.186445] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1190.190951] LustreError: 34978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1190.781945] Lustre: server umount lustre-MDT0001 complete [ 1203.203633] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1206.072594] Key type lgssc unregistered [ 1206.358990] LNet: 42862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.370412] LNetError: 42862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.392917] LNet: Removed LNI 192.168.203.151@tcp [ 1207.164250] Key type .llcrypt unregistered [ 1207.170375] Key type ._llcrypt unregistered [ 1225.474977] Key type ._llcrypt registered [ 1225.481494] Key type .llcrypt registered [ 1225.607363] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 1241.117508] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1241.997276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1242.092360] alg: No test for adler32 (adler32-zlib) [ 1243.179159] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1243.469568] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1245.192527] Key type lgssc registered [ 1246.233921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1280.014745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1280.044489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.283653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1281.338399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1281.476551] Lustre: lustre-MDT0000: new disk, initializing [ 1281.591289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.618052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1286.129683] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1296.723591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.848241] Lustre: 46013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1296.911348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1296.915860] Lustre: Skipped 1 previous similar message [ 1296.978442] Lustre: lustre-MDT0001: new disk, initializing [ 1297.057844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1297.104615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1297.130879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1301.928748] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1311.526662] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1317.295756] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1324.694726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1325.168659] Lustre: lustre-OST0000: new disk, initializing [ 1325.171144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1325.188525] Lustre: 47979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1325.294633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.224789] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1331.831874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1331.843842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1331.962724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1340.459973] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1345.506743] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1345.948129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1350.871675] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1351.053752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1357.796120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1357.797778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.818119] LustreError: Skipped 1 previous similar message [ 1357.822023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1357.876723] Lustre: Skipped 1 previous similar message [ 1360.182980] Lustre: server umount lustre-OST0000 complete [ 1368.037915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1368.043702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1368.053422] Lustre: Skipped 1 previous similar message [ 1368.065856] Lustre: Skipped 2 previous similar messages [ 1371.645376] LustreError: 49520:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000086bfe9df) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1371.963953] Lustre: server umount lustre-MDT0000 complete [ 1376.525989] LustreError: 46814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776652 with bad export cookie 3881967991126538144 [ 1376.527910] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1376.536961] LustreError: 46814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1376.895785] Lustre: server umount lustre-MDT0001 complete [ 1387.751574] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 06:24:22 (1788776662) [ 1395.632808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.162507] LustreError: 50439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.183865] LustreError: 50439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1396.249667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1400.475570] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1401.315294] LustreError: 50440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.447114] LustreError: 50439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1410.184837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.593583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1415.229794] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1424.814702] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1431.636506] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1438.811904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1439.235781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.892847] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1454.919474] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1459.698834] LustreError: 52967:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1460.827859] Lustre: Modifying parameter general.timeout=1 in log params [ 1462.735328] Lustre: Modifying parameter general.timeout=3 in log params [ 1462.745816] Lustre: Skipped 1 previous similar message [ 1465.383109] Lustre: Modifying parameter general.timeout=6 in log params [ 1465.386685] Lustre: Skipped 2 previous similar messages [ 1469.710427] Lustre: Modifying parameter general.timeout=11 in log params [ 1469.715255] Lustre: Skipped 4 previous similar messages [ 1477.724774] Lustre: Modifying parameter general.timeout=21 in log params [ 1477.728938] Lustre: Skipped 9 previous similar messages [ 1494.359051] Lustre: Modifying parameter general.timeout=40 in log params [ 1494.364260] Lustre: Skipped 18 previous similar messages [ 1527.447732] Lustre: Modifying parameter general.timeout=73 in log params [ 1527.454376] Lustre: Skipped 32 previous similar messages [ 1594.445599] Lustre: Disabling parameter general.timeout= in log params [ 1594.454702] Lustre: Skipped 56 previous similar messages [ 1598.005960] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 06:27:53 (1788776873) [ 1600.668413] Lustre: 59914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1602.683554] Lustre: 60026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1604.687675] Lustre: 60125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1608.713572] Lustre: 60335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1608.728192] Lustre: 60335:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1641.702953] Lustre: 61789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1641.716180] Lustre: 61789:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1641.756694] LustreError: 61789:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1647.288081] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 06:28:41 (1788776921) [ 1661.410026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1661.429956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.450423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1665.847108] Lustre: server umount lustre-MDT0000 complete [ 1667.556171] LustreError: 50440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1667.590500] LustreError: 50440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1669.423198] LustreError: 57777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776945 with bad export cookie 3881967991126539026 [ 1669.435300] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1669.892242] Lustre: server umount lustre-MDT0001 complete [ 1684.613368] Lustre: server umount lustre-OST0000 complete [ 1715.573176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.859676] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1716.159209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1720.943594] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1729.670275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.871425] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1729.918805] Lustre: 64317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1735.416361] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1743.233920] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1748.656921] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1756.153935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1756.331645] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1756.680522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1756.692022] Lustre: Skipped 1 previous similar message [ 1763.126042] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1770.981630] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1773.052678] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1773.062795] Lustre: Skipped 23 previous similar messages [ 1773.113695] Lustre: 66885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1781.218081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1781.228936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.246990] Lustre: Skipped 2 previous similar messages [ 1781.255780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.263338] Lustre: Skipped 3 previous similar messages [ 1786.814771] Lustre: server umount lustre-OST0000 complete [ 1791.972047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.975344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1791.976901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.976947] Lustre: Skipped 1 previous similar message [ 1791.989855] Lustre: Skipped 2 previous similar messages [ 1791.998899] LustreError: Skipped 1 previous similar message [ 1796.775244] Lustre: server umount lustre-MDT0000 complete [ 1797.089103] LustreError: 64325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.098940] LustreError: 64325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1800.160586] LustreError: 64310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777076 with bad export cookie 3881967991126542106 [ 1800.172871] LustreError: 64310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1800.180801] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1800.477801] Lustre: server umount lustre-MDT0001 complete [ 1810.983079] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1813.988614] Key type lgssc unregistered [ 1814.218161] LNet: 68096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.228038] LNetError: 68096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.251982] LNet: Removed LNI 192.168.203.151@tcp [ 1815.170576] Key type .llcrypt unregistered [ 1815.172775] Key type ._llcrypt unregistered [ 1839.511773] Key type ._llcrypt registered [ 1839.515692] Key type .llcrypt registered [ 1839.673405] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1841.004884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1841.028200] alg: No test for adler32 (adler32-zlib) [ 1842.171489] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1842.360989] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1844.048268] Key type lgssc registered [ 1845.217737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1852.299297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1852.321767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.772916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.554599] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1863.140407] LustreError: 69507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1863.154523] LustreError: 69507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1865.557418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.049180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1869.967709] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1877.118214] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1882.505393] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1889.289087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.698346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.455872] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1906.297612] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1914.797607] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1924.581233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.590152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1924.608046] Lustre: Skipped 1 previous similar message [ 1924.625023] Lustre: Skipped 2 previous similar messages [ 1929.698632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.707986] Lustre: Skipped 2 previous similar messages [ 1929.971016] Lustre: server umount lustre-MDT0000 complete [ 1933.968076] LustreError: 71358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777210 with bad export cookie 12715646907705847417 [ 1933.986542] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.441380] Lustre: server umount lustre-MDT0001 complete [ 1950.028669] Lustre: server umount lustre-OST0000 complete [ 1978.406984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.741284] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1978.797244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1979.157245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.862693] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1994.225967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1994.363282] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1994.418867] Lustre: 74379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1994.959986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2000.523266] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2008.491647] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2014.806747] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2023.223400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2023.462859] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2023.927376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.011954] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2040.890808] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2042.873538] Lustre: 76959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2045.208710] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2052.067273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2052.076606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.086148] Lustre: Skipped 1 previous similar message [ 2052.090995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2052.578417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2057.049269] Lustre: server umount lustre-OST0000 complete [ 2061.285043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2061.295550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.312797] Lustre: Skipped 1 previous similar message [ 2061.321575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2061.326928] Lustre: Skipped 1 previous similar message [ 2067.280034] Lustre: server umount lustre-MDT0000 complete [ 2067.954054] LustreError: 74392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.984544] LustreError: 74392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2071.655390] LustreError: 75126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777347 with bad export cookie 12715646907705848264 [ 2071.661667] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2071.667882] LustreError: 75126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2072.207368] Lustre: server umount lustre-MDT0001 complete [ 2085.349685] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2089.164409] Key type lgssc unregistered [ 2089.562683] LNet: 78172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.568379] LNetError: 78172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.612162] LNet: Removed LNI 192.168.203.151@tcp [ 2091.575281] Key type .llcrypt unregistered [ 2091.578515] Key type ._llcrypt unregistered [ 2118.778644] Key type ._llcrypt registered [ 2118.782454] Key type .llcrypt registered [ 2118.920212] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2120.013964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2120.101267] alg: No test for adler32 (adler32-zlib) [ 2121.310947] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2121.578559] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2123.297844] Key type lgssc registered [ 2124.409236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.852496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.880327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.485518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.146510] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2144.749199] LustreError: 79593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.782279] LustreError: 79593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2148.303671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.938033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2153.459047] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2162.229626] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2167.674742] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2174.142515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2174.824044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.849526] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2190.940459] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2200.276500] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2201.289320] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2202.643959] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2204.880135] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2204.885462] Lustre: Skipped 1 previous similar message [ 2209.807548] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2209.816542] Lustre: Skipped 4 previous similar messages [ 2218.580119] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2218.590693] Lustre: Skipped 7 previous similar messages [ 2234.974484] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2234.978402] Lustre: Skipped 16 previous similar messages [ 2267.533274] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2267.555512] Lustre: Skipped 32 previous similar messages [ 2309.728647] Lustre: 87701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2312.425614] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 06:39:47 (1788777587) [ 2314.313781] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2315.923041] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 06:39:51 (1788777591) [ 2321.380253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.389182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2321.408485] Lustre: Skipped 1 previous similar message [ 2321.429844] Lustre: Skipped 1 previous similar message [ 2326.500304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2326.507721] Lustre: Skipped 1 previous similar message [ 2331.636783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2331.642931] Lustre: Skipped 1 previous similar message [ 2334.176174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2334.352751] Lustre: server umount lustre-OST0000 complete [ 2341.859576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.865029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2341.883928] Lustre: Skipped 1 previous similar message [ 2345.106194] Lustre: server umount lustre-MDT0000 complete [ 2346.980376] LustreError: 79593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.002326] LustreError: 79593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2349.147413] LustreError: 82186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777625 with bad export cookie 15371694346349045795 [ 2349.149752] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2349.156565] LustreError: 82186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2349.553203] Lustre: server umount lustre-MDT0001 complete [ 2360.645495] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2363.267129] Key type lgssc unregistered [ 2363.570871] LNet: 88951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.582658] LNetError: 88951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.612377] LNet: Removed LNI 192.168.203.151@tcp [ 2364.565399] Key type .llcrypt unregistered [ 2364.568473] Key type ._llcrypt unregistered [ 2369.363970] Key type ._llcrypt registered [ 2369.366158] Key type .llcrypt registered [ 2369.483862] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2374.598540] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules [ 2375.751312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2375.781305] alg: No test for adler32 (adler32-zlib) [ 2377.003932] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2377.360732] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2379.003262] Key type lgssc registered [ 2380.812388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2382.474821] LustreError: 90197:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2400.733330] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2405.275922] LustreError: 90197:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2410.165479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2410.195261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.749419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.395265] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2421.223371] LustreError: 91221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.242318] LustreError: 91221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2426.229275] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 06:41:41 (1788777701) [ 2426.348667] LustreError: 91222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.461434] LustreError: 91221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.854647] LustreError: 91254:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2431.888552] LustreError: 91254:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2431.911038] LustreError: 91254:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2432.589148] Lustre: server umount lustre-MDT0000 complete [ 2447.885767] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2450.748939] Key type lgssc unregistered [ 2451.066960] LNet: 92749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.079854] LNetError: 92749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2451.097852] LNet: Removed LNI 192.168.203.151@tcp [ 2451.928448] Key type .llcrypt unregistered [ 2451.935040] Key type ._llcrypt unregistered [ 2469.628407] Key type ._llcrypt registered [ 2469.633781] Key type .llcrypt registered [ 2469.812860] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2470.851904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2470.865346] alg: No test for adler32 (adler32-zlib) [ 2472.125325] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2472.392529] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2474.072197] Key type lgssc registered [ 2475.829886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.937933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2485.951526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.807095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.718516] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2497.005307] LustreError: 93962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.061904] LustreError: 93962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2502.114818] LustreError: 93963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.640840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.214321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2513.492321] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2524.691988] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2531.235916] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2538.967852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2539.749287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.577894] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2556.941633] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2611.528472] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 06:44:46 (1788777886) [ 2613.419123] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2615.075806] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 06:44:50 (1788777890) [ 2619.363863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2619.370060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.381562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2623.970844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2623.977597] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.675902] Lustre: server umount lustre-MDT0000 complete [ 2629.096475] LustreError: 94704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.121450] LustreError: 94704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2629.482948] LustreError: 93943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777905 with bad export cookie 4245374287609613838 [ 2629.492838] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2629.497912] LustreError: 93943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2629.797264] Lustre: server umount lustre-MDT0001 complete [ 2644.713219] Lustre: server umount lustre-OST0000 complete [ 2659.518674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.108325] LustreError: 98262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.424245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.443249] LustreError: 98263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.862631] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2675.682310] LustreError: 98262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.715623] LustreError: 98262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2676.220704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.941477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2682.403397] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2691.500499] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2696.738927] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2711.094627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.386571] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2711.399429] LustreError: 98252:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2711.452546] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2711.467986] LustreError: 100347:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2711.489614] LustreError: 100347:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2711.503619] LustreError: 100347:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2711.649218] Lustre: server umount lustre-OST0000 complete [ 2711.655837] LustreError: 100347:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2720.961750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2721.167064] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2721.179130] LustreError: 98251:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2721.187786] LustreError: 100819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.215670] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2721.229627] LustreError: 100815:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2721.236331] LustreError: 100815:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2721.240845] LustreError: 100815:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2721.343238] Lustre: server umount lustre-OST0000 complete [ 2721.347302] LustreError: 100815:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2732.806090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2733.121622] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2733.127676] Lustre: Found index 0 for lustre-OST0000, updating log [ 2733.140378] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2733.252586] Lustre: lustre-OST0000: new disk, initializing [ 2733.260379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2733.267566] Lustre: 101348:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2733.574405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2740.653714] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2744.791862] Lustre: Failing over lustre-OST0000 [ 2745.040350] Lustre: server umount lustre-OST0000 complete [ 2749.157666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.172581] Lustre: Skipped 1 previous similar message [ 2749.184328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.193409] Lustre: Skipped 3 previous similar messages [ 2751.267694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.279721] Lustre: Skipped 1 previous similar message [ 2755.589223] Lustre: server umount lustre-MDT0000 complete [ 2756.582824] LustreError: 98262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.602817] LustreError: 98262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2759.547660] LustreError: 98242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778035 with bad export cookie 4245374287609616582 [ 2759.549870] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2759.567186] LustreError: 98242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2759.971522] Lustre: server umount lustre-MDT0001 complete [ 2769.826988] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 06:47:24 (1788778044) [ 2789.497834] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2800.066831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.479290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.716911] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2813.174459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.313879] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2822.769576] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2830.816486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2839.107635] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2851.137919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.284621] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2851.443646] Lustre: lustre-OST0001: new disk, initializing [ 2851.454694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2851.468956] Lustre: 106012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2852.002103] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2852.016217] Lustre: Skipped 2 previous similar messages [ 2860.323639] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2861.163453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2861.174599] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2861.300794] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2868.246251] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2875.862704] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2882.027716] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.034045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2882.060811] Lustre: Skipped 3 previous similar messages [ 2882.065666] Lustre: Skipped 2 previous similar messages [ 2886.805102] Lustre: server umount lustre-MDT0001 complete [ 2887.147782] LustreError: 103691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.159810] LustreError: 103691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2896.524876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2896.758976] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2896.769878] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2896.859624] Lustre: 103682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2898.217778] LustreError: 107547:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2898.231739] LustreError: 107547:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2898.241909] LustreError: 103687:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2898.381896] LustreError: 107555:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.151@tcp: cfg command failed: rc = -17 [ 2898.394479] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2898.394479] [ 2902.639458] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2903.042981] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2913.757664] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 06:49:48 (1788778188) [ 2936.568351] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2960.774546] Lustre: 109632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2989.725863] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2993.973179] Lustre: 110768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3086.818786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3086.845120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.870994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3086.874206] Lustre: Skipped 1 previous similar message [ 3088.674771] Lustre: server umount lustre-MDT0000 complete [ 3092.451554] LustreError: 104662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.488614] LustreError: 104662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3093.041558] LustreError: 106733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778369 with bad export cookie 4245374287609617576 [ 3093.051741] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3093.060790] LustreError: 106733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3093.486826] Lustre: server umount lustre-MDT0001 complete [ 3108.153670] Lustre: server umount lustre-OST0000 complete [ 3123.215091] Lustre: server umount lustre-OST0001 complete [ 3130.155159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3141.140492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3150.098665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3160.572458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3184.499382] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3195.798242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.842506] Lustre: lustre-MDT0000: reset Object Index mappings [ 3196.575993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3196.583971] Lustre: Skipped 1 previous similar message [ 3201.094505] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3210.832469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.866296] Lustre: lustre-MDT0001: reset Object Index mappings [ 3211.236087] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3211.239180] Lustre: Skipped 4 previous similar messages [ 3219.897290] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3223.713434] Lustre: 115873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3232.316322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.098476] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3240.219861] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3241.210065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3250.299738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3256.328295] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3256.334607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3257.841492] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3266.048634] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3270.493156] Lustre: 117750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3327.542881] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 06:56:42 (1788778602) [ 3353.579964] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.597845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3353.601967] Lustre: Skipped 5 previous similar messages [ 3353.618124] Lustre: Skipped 3 previous similar messages [ 3366.368145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3366.627596] Lustre: server umount lustre-MDT0000 complete [ 3368.933608] LustreError: 117765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.954666] LustreError: 117765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3370.446929] LustreError: 114705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778646 with bad export cookie 4245374287609664490 [ 3370.454280] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3370.469671] LustreError: 114705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3374.052369] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3374.073621] Lustre: Skipped 2 previous similar messages [ 3377.038603] Lustre: server umount lustre-MDT0001 complete [ 3387.373099] Lustre: server umount lustre-OST0000 complete [ 3406.304683] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3406.785213] Lustre: server umount lustre-OST0001 complete [ 3416.550942] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 3427.204407] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3472.227116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.480179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3472.511041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3472.634625] Lustre: lustre-MDT0000: new disk, initializing [ 3472.725809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.731773] Lustre: Skipped 3 previous similar messages [ 3472.759313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3477.754273] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3485.060396] Lustre: server umount lustre-MDT0000 complete [ 3495.185806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3495.512427] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3495.524448] Lustre: 122027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3495.536495] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3500.496345] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3506.686771] Lustre: server umount lustre-MDT0000 complete [ 3517.035209] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 06:59:52 (1788778792) [ 3519.299524] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3521.639808] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 06:59:56 (1788778796) [ 3542.070531] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3555.752307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3561.810785] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3573.833762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.999064] Lustre: 124081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3574.097792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3574.101843] Lustre: Skipped 1 previous similar message [ 3574.200859] Lustre: lustre-MDT0001: new disk, initializing [ 3574.542849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3574.571697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3581.687753] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3588.355127] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3596.972105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3597.543179] Lustre: lustre-OST0000: new disk, initializing [ 3597.550274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3597.569753] Lustre: 125674:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3598.946502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3598.975890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3599.182889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3607.296244] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3620.711888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.012277] Lustre: lustre-OST0001: new disk, initializing [ 3621.017674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3621.046599] Lustre: 126545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3622.841997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3622.867193] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3622.981676] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3631.026244] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3643.499484] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3648.012339] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4048.559288] Lustre: 132970:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4048.567597] Lustre: 132970:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4048.574434] Lustre: 132970:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4048.580955] Lustre: 132970:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4048.592579] Lustre: 132970:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4048.598487] Lustre: 132970:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4356.792919] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:13:51 (1788779631) [ 4373.473665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4373.480068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.480857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.480861] Lustre: Skipped 12 previous similar messages [ 4373.523762] Lustre: Skipped 3 previous similar messages [ 4374.126439] Lustre: server umount lustre-MDT0000 complete [ 4378.125817] LustreError: 124073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779654 with bad export cookie 4245374287609733573 [ 4378.141034] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.150932] LustreError: 124073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4378.600524] LustreError: 124088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.600780] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.614985] LustreError: 124088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4378.636447] Lustre: Skipped 1 previous similar message [ 4383.715385] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4383.727395] Lustre: Skipped 6 previous similar messages [ 4384.595789] Lustre: server umount lustre-MDT0001 complete [ 4395.326818] Lustre: server umount lustre-OST0000 complete [ 4413.408157] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4413.623721] Lustre: server umount lustre-OST0001 complete [ 4420.671810] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 4430.450615] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 4464.070539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.322900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4464.373483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4464.460559] Lustre: lustre-MDT0000: new disk, initializing [ 4464.522703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.531233] Lustre: Skipped 5 previous similar messages [ 4464.542501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4468.649905] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4477.565123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.712273] Lustre: 136026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4477.759586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4477.765375] Lustre: Skipped 1 previous similar message [ 4477.840766] Lustre: lustre-MDT0001: new disk, initializing [ 4477.962653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4477.974801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4481.609327] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4489.735334] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4494.935332] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4501.242223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4501.465121] Lustre: lustre-OST0000: new disk, initializing [ 4501.468513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4501.473837] Lustre: 137992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4501.553244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4501.563801] Lustre: Skipped 1 previous similar message [ 4503.044647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4503.055621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4503.187829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4506.741234] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4515.121602] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4524.834253] Lustre: lustre-MDD0000: changelog on [ 4527.420269] Lustre: lustre-MDD0001: changelog on [ 6639.865175] Lustre: lustre-MDD0001: changelog off [ 6642.086386] Lustre: 139449:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6643.044432] Lustre: lustre-MDD0000: changelog off [ 6653.434963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.456760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6653.458469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6658.549888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6658.561981] Lustre: Skipped 2 previous similar messages [ 6663.654254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6664.160775] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6664.473683] Lustre: server umount lustre-OST0000 complete [ 6669.281993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6669.287712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.306428] Lustre: Skipped 1 previous similar message [ 6673.889554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.896131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.909430] Lustre: Skipped 2 previous similar messages [ 6675.752473] Lustre: server umount lustre-MDT0000 complete [ 6679.013273] LustreError: 136039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6679.038862] LustreError: 136039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6679.614545] LustreError: 136021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781955 with bad export cookie 4245374287610554477 [ 6679.640211] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6679.642133] LustreError: 136021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6680.121169] Lustre: server umount lustre-MDT0001 complete [ 6693.333888] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 6697.285537] Key type lgssc unregistered [ 6697.724615] LNet: 147169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6697.734617] LNetError: 147169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6697.778736] LNet: Removed LNI 192.168.203.151@tcp [ 6698.855204] Key type .llcrypt unregistered [ 6698.857297] Key type ._llcrypt unregistered [ 6727.313251] Key type ._llcrypt registered [ 6727.316764] Key type .llcrypt registered [ 6727.455584] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 6747.650694] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 6748.984542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6749.109940] alg: No test for adler32 (adler32-zlib) [ 6750.325830] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 6750.642927] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 6752.360161] Key type lgssc registered [ 6754.153103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6798.974308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6798.989111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.317750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6800.375271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6800.514420] Lustre: lustre-MDT0000: new disk, initializing [ 6800.725585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6800.766872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6806.218373] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6817.848872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.007754] Lustre: 150372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6818.066777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6818.071090] Lustre: Skipped 1 previous similar message [ 6818.133719] Lustre: lustre-MDT0001: new disk, initializing [ 6818.226625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6818.288308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6818.305062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6822.902741] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6832.315659] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6839.261138] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6846.745454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6846.970073] Lustre: lustre-OST0000: new disk, initializing [ 6846.973982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6846.979171] Lustre: 152339:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6847.074779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6854.795358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6854.831754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6855.018897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6856.320798] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6868.532544] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6874.534368] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_module obdecho/obdecho [ 6876.414502] LustreError: 153343:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6876.423452] LustreError: 153343:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6878.253650] LustreError: 153440:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6895.079952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6895.090224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.126591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6896.111234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6896.119622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6896.133064] Lustre: Skipped 1 previous similar message [ 6896.149504] Lustre: Skipped 1 previous similar message [ 6898.392439] Lustre: server umount lustre-MDT0000 complete [ 6902.738808] LustreError: 150364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782178 with bad export cookie 16558201529417999056 [ 6902.745411] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6902.756448] LustreError: 150364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6903.181579] Lustre: server umount lustre-MDT0001 complete [ 6918.562726] Lustre: server umount lustre-OST0000 complete [ 6943.904308] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 6956.666907] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7000.261308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7000.548781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7000.594600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7000.716564] Lustre: lustre-MDT0000: new disk, initializing [ 7000.815161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7000.838346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7006.087657] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7017.759265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7017.934459] Lustre: 157161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7018.069158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7018.079083] Lustre: Skipped 1 previous similar message [ 7018.198960] Lustre: lustre-MDT0001: new disk, initializing [ 7018.289300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7018.325279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7018.335888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7023.669671] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7034.160863] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7041.336908] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7049.964447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7050.246301] Lustre: lustre-OST0000: new disk, initializing [ 7050.248595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7050.255966] Lustre: 159128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7050.344182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7056.483518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7056.502549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7056.654810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7058.054416] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7069.139413] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7074.285348] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7074.539466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7079.750747] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7080.049558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7082.468170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.488625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7082.496803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7082.500351] LustreError: Skipped 1 previous similar message [ 7082.520885] Lustre: Skipped 2 previous similar messages [ 7087.591973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7087.605967] Lustre: Skipped 1 previous similar message [ 7088.736358] Lustre: server umount lustre-OST0000 complete [ 7095.266619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7095.286348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7095.328597] Lustre: Skipped 1 previous similar message [ 7095.335246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.118593] Lustre: server umount lustre-MDT0000 complete [ 7102.964411] LustreError: 157172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.071154] LustreError: 157172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7107.581659] LustreError: 157153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782383 with bad export cookie 16558201529418000022 [ 7107.598934] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7107.610521] LustreError: 157153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7108.027334] Lustre: server umount lustre-MDT0001 complete [ 7113.062575] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 07:59:48 (1788782388) [ 7116.114847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7118.906209] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 07:59:53 (1788782393) [ 7128.154755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.563553] LustreError: 161544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7128.675099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7133.679975] LustreError: 161545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.346288] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7138.786215] LustreError: 161544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7143.917440] LustreError: 161545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.503913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.575746] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7163.089562] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7169.779033] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7178.223008] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7178.603898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7178.611355] Lustre: Skipped 1 previous similar message [ 7185.390538] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7193.386280] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7207.396771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7207.406867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7207.426090] Lustre: Skipped 1 previous similar message [ 7207.437896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7207.445946] Lustre: Skipped 2 previous similar messages [ 7211.335576] Lustre: server umount lustre-MDT0000 complete [ 7213.547206] LustreError: 161545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7213.584038] LustreError: 161545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7215.432164] LustreError: 161526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782491 with bad export cookie 16558201529418000897 [ 7215.442084] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7215.453368] LustreError: 161526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7218.659821] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7218.686869] Lustre: Skipped 2 previous similar messages [ 7222.088659] Lustre: server umount lustre-MDT0001 complete [ 7229.564921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7243.269838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.526482] LustreError: 165419:0:(llog.c:1655:llog_backup()) MGC192.168.203.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7243.532710] Lustre: 165419:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7245.283930] LustreError: 165424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.308612] LustreError: 165424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7281.121964] LustreError: 165426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7281.148513] LustreError: 165426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7304.164452] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7304.168713] Lustre: 165419:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7345.140791] LustreError: 165426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.150558] LustreError: 165426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 7345.155691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7349.803177] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7350.249368] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7359.084032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7360.291453] LustreError: 166161:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7360.302482] LustreError: 166161:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000abe1e117 x1875673901828736/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788782636 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7360.338068] LustreError: 166161:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7365.918778] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7366.126782] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7373.766590] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7375.675668] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7381.949928] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7383.725931] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7406.350386] Lustre: server umount lustre-MDT0000 complete [ 7407.076280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7407.078574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.094734] Lustre: Skipped 1 previous similar message [ 7410.550795] LustreError: 164360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782686 with bad export cookie 16558201529418007358 [ 7410.554660] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7410.568227] LustreError: 164360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7411.227031] Lustre: server umount lustre-MDT0001 complete [ 7425.604482] Lustre: server umount lustre-OST0000 complete [ 7451.539158] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7461.823399] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7503.420629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7503.670701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7503.721989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7503.846868] Lustre: lustre-MDT0000: new disk, initializing [ 7503.939035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.953611] Lustre: Skipped 1 previous similar message [ 7503.983324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7509.652399] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7522.452977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7522.640510] Lustre: 170522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7523.023790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7528.164593] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7538.476527] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7545.574679] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7553.057250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7553.449796] Lustre: lustre-OST0000: new disk, initializing [ 7553.454604] Lustre: Skipped 1 previous similar message [ 7553.462733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7553.470438] Lustre: Skipped 2 previous similar messages [ 7553.480369] Lustre: 172492:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7559.276560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7559.282476] Lustre: Skipped 1 previous similar message [ 7559.287199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7559.510524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7561.638959] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7572.353654] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7577.157554] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7577.434362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7582.266984] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7582.469811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7585.249443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7585.258606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7585.264706] LustreError: Skipped 1 previous similar message [ 7585.266868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7585.266875] Lustre: Skipped 7 previous similar messages [ 7585.301916] Lustre: Skipped 2 previous similar messages [ 7590.825288] Lustre: server umount lustre-OST0000 complete [ 7600.918654] Lustre: server umount lustre-MDT0000 complete [ 7604.731372] LustreError: 170516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782880 with bad export cookie 16558201529418024193 [ 7604.744454] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7604.745800] LustreError: 170516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7605.149903] Lustre: server umount lustre-MDT0001 complete [ 7614.867747] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:08:10 (1788782890) [ 7621.754344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7622.237372] LustreError: 174902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7622.258430] LustreError: 174902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 7626.907266] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7635.414708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7640.251441] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7647.714974] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7653.634381] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7660.435851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7667.723553] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7676.191676] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7688.982517] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:09:24 (1788782964) [ 7693.793054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7693.809557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.832107] Lustre: Skipped 2 previous similar messages [ 7693.840914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7693.850663] Lustre: Skipped 7 previous similar messages [ 7699.220777] Lustre: server umount lustre-OST0000 complete [ 7717.344349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7717.517337] Lustre: server umount lustre-MDT0000 complete [ 7721.443048] LustreError: 174882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782997 with bad export cookie 16558201529418025075 [ 7721.452092] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7721.463653] LustreError: 174882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7721.881189] Lustre: server umount lustre-MDT0001 complete [ 7732.894576] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7735.322226] Key type lgssc unregistered [ 7735.629874] LNet: 178631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7735.640068] LNetError: 178631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7735.657836] LNet: Removed LNI 192.168.203.151@tcp [ 7736.509904] Key type .llcrypt unregistered [ 7736.513344] Key type ._llcrypt unregistered [ 7754.735209] Key type ._llcrypt registered [ 7754.738181] Key type .llcrypt registered [ 7754.846384] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7755.545119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7755.570138] alg: No test for adler32 (adler32-zlib) [ 7756.613571] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 7756.780978] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7758.424150] Key type lgssc registered [ 7759.485841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7767.316222] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7767.340831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7782.944363] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7788.066187] LustreError: 179871:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.151@tcp: failed processing log, type 4: rc = -110 [ 7793.184598] LustreError: 179871:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7793.208482] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7793.217683] LustreError: MGC192.168.203.151@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7793.235555] LustreError: 179871:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7793.245712] LustreError: 179871:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7793.258690] LustreError: 179871:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7793.320499] Lustre: server umount lustre-OST0000 complete [ 7793.326732] LustreError: 179871:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7802.192698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.651044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7807.621853] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7813.099921] LustreError: 180370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7813.130394] LustreError: 180370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7817.386837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.816679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7822.488981] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7830.744178] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7837.348665] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7843.920815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7844.064087] LustreError: 182187:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7844.073660] Lustre: 182187:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.151@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7844.210156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7850.323717] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7857.832380] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7862.244473] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7862.250165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7862.254140] Lustre: Skipped 2 previous similar messages [ 7862.266033] Lustre: Skipped 2 previous similar messages [ 7866.542167] Lustre: server umount lustre-MDT0000 complete [ 7867.363632] LustreError: 180367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7867.401475] LustreError: 180367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7870.249155] LustreError: 180352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783146 with bad export cookie 3717528387973983119 [ 7870.255193] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7870.269983] LustreError: 180352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7870.450223] LustreError: 181101:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000a6e3ef8 x1875674956591616/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7870.602094] Lustre: server umount lustre-MDT0001 complete [ 7885.093334] Lustre: server umount lustre-OST0000 complete [ 7896.183367] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7906.747894] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7942.724173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7942.917788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7942.971704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7943.133327] Lustre: lustre-MDT0000: new disk, initializing [ 7943.251928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7943.279792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7948.128107] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7959.287837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7959.423202] Lustre: 185715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7959.478613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7959.482218] Lustre: Skipped 1 previous similar message [ 7959.556245] Lustre: lustre-MDT0001: new disk, initializing [ 7959.644424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7959.707115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7959.716690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7963.979391] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7974.470951] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7980.337154] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7987.156861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.444201] Lustre: lustre-OST0000: new disk, initializing [ 7987.447382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7987.452750] Lustre: 187683:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7987.547970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7993.440065] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7996.973464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7996.990423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7997.110123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8002.697931] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8007.535643] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8007.751360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8012.938805] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8013.110751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8017.893706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.900361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8017.916038] Lustre: Skipped 1 previous similar message [ 8017.916298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8020.971316] Lustre: server umount lustre-OST0000 complete [ 8026.082785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8026.087126] LustreError: Skipped 1 previous similar message [ 8026.093054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.107533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8026.110331] Lustre: Skipped 1 previous similar message [ 8028.133119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.135461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8028.172577] Lustre: Skipped 1 previous similar message [ 8033.252742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8033.261369] Lustre: Skipped 1 previous similar message [ 8039.392487] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8039.682884] Lustre: server umount lustre-MDT0000 complete [ 8043.228210] LustreError: 185706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783319 with bad export cookie 3717528387973983840 [ 8043.239780] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8043.257373] LustreError: 185706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8043.659470] Lustre: server umount lustre-MDT0001 complete [ 8051.941223] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:15:27 (1788783327) [ 8068.788286] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8071.533903] Key type lgssc unregistered [ 8071.851944] LNet: 190627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8071.864063] LNetError: 190627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8071.887554] LNet: Removed LNI 192.168.203.151@tcp [ 8072.688206] Key type .llcrypt unregistered [ 8072.691141] Key type ._llcrypt unregistered [ 8090.168389] Key type ._llcrypt registered [ 8090.173412] Key type .llcrypt registered [ 8090.264966] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8091.227156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8091.255378] alg: No test for adler32 (adler32-zlib) [ 8092.328067] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 8092.639286] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8094.411036] Key type lgssc registered [ 8095.698749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8105.646601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8105.682393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.262873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8112.082851] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8117.732958] LustreError: 191848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.750538] LustreError: 191848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8121.705530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.113089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8126.984328] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8134.462103] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8140.915729] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8148.925192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8149.180523] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8149.188578] LustreError: 193712:0:(llog.c:1696:llog_backup()) MGC192.168.203.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8149.199261] Lustre: 193712:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.151@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8149.343390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8155.894101] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8162.504364] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8167.907623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.914522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8167.930175] Lustre: Skipped 2 previous similar messages [ 8167.944391] Lustre: Skipped 2 previous similar messages [ 8168.995316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8169.015801] Lustre: Skipped 2 previous similar messages [ 8170.826715] Lustre: server umount lustre-MDT0000 complete [ 8174.053586] LustreError: 191849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.070799] LustreError: 191849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8174.310675] LustreError: 191829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783450 with bad export cookie 5189023707605144137 [ 8174.315913] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8174.325956] LustreError: 191829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8175.200478] LustreError: 192577:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000049e2c8c6 x1875675308911872/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8175.400244] Lustre: server umount lustre-MDT0001 complete [ 8188.338131] Lustre: server umount lustre-OST0000 complete [ 8199.305317] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8211.120363] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8247.740480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.966039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8248.004341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8248.082721] Lustre: lustre-MDT0000: new disk, initializing [ 8248.178269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.195853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8252.606269] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8262.511411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.634034] Lustre: 197236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8262.678322] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8262.681147] Lustre: Skipped 1 previous similar message [ 8262.684285] LustreError: 198019:0:(llog.c:1696:llog_backup()) MGC192.168.203.151@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8262.690795] LustreError: 198019:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8262.695480] Lustre: 198019:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.151@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8262.710192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8262.713267] Lustre: Skipped 1 previous similar message [ 8262.767660] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8262.780070] Lustre: lustre-MDT0001: new disk, initializing [ 8262.842250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8262.880593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8262.896470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8266.772429] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8274.823980] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8280.488256] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8287.529314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8287.829879] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8287.836786] Lustre: Skipped 1 previous similar message [ 8287.843977] LustreError: 199181:0:(llog.c:1696:llog_backup()) MGC192.168.203.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8287.861323] LustreError: 199181:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8287.866786] Lustre: 199181:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.151@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8287.888841] Lustre: 199181:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8287.941117] Lustre: lustre-OST0000: new disk, initializing [ 8287.948577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8287.962191] Lustre: 199204:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8288.143311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.504552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8294.517167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8294.583616] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8294.623190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8305.075810] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8310.352715] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8310.627289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8315.550757] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8315.748305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8320.481125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8320.483761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.492305] LustreError: Skipped 1 previous similar message [ 8320.516606] Lustre: Skipped 1 previous similar message [ 8320.520846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.534239] Lustre: Skipped 1 previous similar message [ 8323.910666] Lustre: server umount lustre-OST0000 complete [ 8329.700691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8329.711157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.734750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8333.721760] Lustre: server umount lustre-MDT0000 complete [ 8335.846336] LustreError: 197248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8335.862662] LustreError: 197248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8337.298785] LustreError: 197229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783613 with bad export cookie 5189023707605144858 [ 8337.301820] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8337.315400] LustreError: 197229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8337.600410] Lustre: server umount lustre-MDT0001 complete [ 8346.103125] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:20:21 (1788783621) [ 8366.729433] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8376.034472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.581421] LustreError: 202346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.641217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8380.742365] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8381.929220] LustreError: 202347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8388.912604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.723844] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8396.484721] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8402.145079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8409.337257] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8417.640062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8417.788955] Lustre: lustre-OST0001: new disk, initializing [ 8417.794461] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8417.804749] Lustre: 204655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8417.906263] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8417.911078] Lustre: Skipped 2 previous similar messages [ 8423.714712] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8425.012354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8425.024690] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8425.127351] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8434.008843] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8443.011897] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8445.174869] Lustre: DEBUG MARKER: ADD OST3 [ 8451.223673] loop: module loaded [ 8451.273885] loop0: detected capacity change from 0 to 204800000 [ 8451.531648] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8451.545966] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8451.554841] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8458.029094] loop0: detected capacity change from 0 to 204800000 [ 8458.729414] Lustre: DEBUG MARKER: STOP OST3 [ 8464.139736] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8464.397966] Lustre: lustre-OST0003: new disk, initializing [ 8464.406970] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8464.421986] Lustre: 206661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8465.922684] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8466.054559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8466.071808] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8466.132888] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8467.830136] Lustre: Failing over lustre-OST0003 [ 8467.936239] LustreError: 206828:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8468.036320] Lustre: server umount lustre-OST0003 complete [ 8470.650814] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8471.027689] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.042085] Lustre: Skipped 2 previous similar messages [ 8471.043068] LustreError: 203830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8471.080113] LustreError: 203830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8479.761872] Lustre: DEBUG MARKER: START OST3 again [ 8486.707550] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8486.998787] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8487.007663] Lustre: Skipped 1 previous similar message [ 8487.033825] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8488.528467] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8488.845110] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8488.848994] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8493.241232] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8514.820063] Lustre: Failing over lustre-OST0003 [ 8515.000822] Lustre: server umount lustre-OST0003 complete [ 8517.617919] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8517.645546] LustreError: 205088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8517.672543] LustreError: 205088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8532.963820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8532.964901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8532.969244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8532.969253] Lustre: Skipped 2 previous similar messages [ 8532.991344] Lustre: Skipped 4 previous similar messages [ 8538.702360] Lustre: server umount lustre-MDT0000 complete [ 8542.442837] LustreError: 205423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783818 with bad export cookie 5189023707605145740 [ 8542.443790] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8542.454384] LustreError: 205423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8548.936978] Lustre: server umount lustre-MDT0001 complete [ 8552.747913] Lustre: server umount lustre-OST0000 complete [ 8556.546440] Lustre: server umount lustre-OST0001 complete [ 8563.380987] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8571.582275] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8612.421309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8612.701755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8612.734269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8612.857550] Lustre: lustre-MDT0000: new disk, initializing [ 8613.038763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8617.871504] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8628.950078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8629.067129] Lustre: 212110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8629.115580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8629.122294] Lustre: Skipped 1 previous similar message [ 8629.208817] Lustre: lustre-MDT0001: new disk, initializing [ 8629.321789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8629.326903] Lustre: Skipped 1 previous similar message [ 8629.356819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8629.368816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8633.908506] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8643.479501] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8649.155877] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8655.804538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8656.054297] Lustre: 214079:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8657.576392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8657.707955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8662.424864] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8672.697598] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8678.075030] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8678.275725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8682.755504] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8682.967878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8688.613538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.616564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8688.620738] Lustre: Skipped 3 previous similar messages [ 8688.620792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8688.624391] Lustre: Skipped 11 previous similar messages [ 8699.360136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8699.529479] Lustre: server umount lustre-OST0000 complete [ 8718.306536] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8718.443587] Lustre: server umount lustre-MDT0000 complete [ 8719.329572] LustreError: 212122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8719.339665] LustreError: 212122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8722.757311] LustreError: 212102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783998 with bad export cookie 5189023707605148036 [ 8722.767250] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8722.774625] LustreError: 212102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8723.111591] Lustre: server umount lustre-MDT0001 complete [ 8727.463396] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:26:42 (1788784002) [ 8745.212040] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8757.777623] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8796.806625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.161890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8797.193736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8797.202676] Lustre: Skipped 1 previous similar message [ 8797.297258] Lustre: lustre-MDT0000: new disk, initializing [ 8797.303612] Lustre: Skipped 1 previous similar message [ 8797.386795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8797.394298] Lustre: Skipped 1 previous similar message [ 8802.404260] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8813.998716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.165791] Lustre: 218694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8814.607944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8819.742850] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8830.728494] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8837.845765] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8846.352389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8846.646326] Lustre: 220666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8848.611074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8848.731435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8854.172562] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8864.460944] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8869.808986] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8870.087876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8875.506910] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8875.789677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8879.592169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8879.594972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8879.597450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8879.597458] Lustre: Skipped 11 previous similar messages [ 8879.605468] LustreError: Skipped 1 previous similar message [ 8879.615015] Lustre: Skipped 3 previous similar messages [ 8892.387970] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8892.638195] Lustre: server umount lustre-OST0000 complete [ 8896.481280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8902.692221] Lustre: server umount lustre-MDT0000 complete [ 8905.198398] LustreError: 218705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8905.226026] LustreError: 218705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8907.188456] LustreError: 218687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784183 with bad export cookie 5189023707605148918 [ 8907.201714] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8907.214224] LustreError: 218687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8907.908771] Lustre: server umount lustre-MDT0001 complete [ 8918.462729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.960416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8918.971392] Lustre: Skipped 4 previous similar messages [ 8923.851652] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8932.089581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.777832] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8944.008787] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8949.904038] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8956.994994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8963.236669] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8969.798196] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8983.026617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8983.034239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8983.057623] Lustre: Skipped 2 previous similar messages [ 8983.063898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8983.075355] Lustre: Skipped 8 previous similar messages [ 8994.272531] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8994.458649] Lustre: server umount lustre-OST0000 complete [ 9012.192116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9012.499829] Lustre: server umount lustre-MDT0000 complete [ 9016.921671] LustreError: 223619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784292 with bad export cookie 5189023707605149800 [ 9016.935032] LustreError: 223619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9016.937685] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9017.301344] Lustre: server umount lustre-MDT0001 complete [ 9107.556232] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9110.428848] Key type lgssc unregistered [ 9110.828597] LNet: 226547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9110.836321] LNetError: 226547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9110.864870] LNet: Removed LNI 192.168.203.151@tcp [ 9111.600480] Key type .llcrypt unregistered [ 9111.609324] Key type ._llcrypt unregistered [ 9137.402323] Key type ._llcrypt registered [ 9137.407098] Key type .llcrypt registered [ 9137.540252] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 9155.940085] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9157.006044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9157.229954] alg: No test for adler32 (adler32-zlib) [ 9158.356973] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 9158.762043] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 9160.552165] Key type lgssc registered [ 9162.109915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9199.897725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9199.924621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9201.251578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9201.296138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9201.458814] Lustre: lustre-MDT0000: new disk, initializing [ 9201.560483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.577676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9206.758399] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9217.990898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9218.158510] Lustre: 229897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9218.259027] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9218.265044] Lustre: Skipped 1 previous similar message [ 9218.368181] Lustre: lustre-MDT0001: new disk, initializing [ 9218.449481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9218.504458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9218.517647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9222.386567] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9231.152933] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9236.843680] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9243.776751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9244.129879] Lustre: lustre-OST0000: new disk, initializing [ 9244.133678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9244.147069] Lustre: 231866:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9244.272566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9250.590561] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9253.469400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9253.476312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9253.615362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9260.649217] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9265.795758] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9266.019619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9270.619548] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9270.915537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9274.341068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9274.342742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9274.343658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9274.374780] Lustre: Skipped 1 previous similar message [ 9274.411903] LustreError: Skipped 1 previous similar message [ 9279.322802] Lustre: server umount lustre-OST0000 complete [ 9284.578507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9284.581556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9284.600303] Lustre: Skipped 1 previous similar message [ 9284.611101] Lustre: Skipped 1 previous similar message [ 9291.827356] Lustre: server umount lustre-MDT0000 complete [ 9294.817932] LustreError: 229908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9294.843094] LustreError: 229908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9295.721904] LustreError: 229887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784571 with bad export cookie 13961836329367327641 [ 9295.740690] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9295.746322] LustreError: 229887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9296.273411] Lustre: server umount lustre-MDT0001 complete [ 9318.564711] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9331.698233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9332.030031] LustreError: 234799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9332.074127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9336.973501] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9337.323862] LustreError: 234800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9342.432961] LustreError: 234799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9347.330370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9347.567286] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9347.701389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9352.499204] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9355.445861] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9362.444754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9362.788517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9370.085251] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9379.055945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9379.212859] Lustre: lustre-OST0001: new disk, initializing [ 9379.223452] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9379.228396] Lustre: 237107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9379.313968] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9385.716136] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9386.550949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9386.570786] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9386.730727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9397.409910] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9408.761868] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9422.520188] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:38:17 (1788784697) [ 9427.940082] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9427.952821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9427.963274] Lustre: Skipped 2 previous similar messages [ 9427.981763] Lustre: Skipped 3 previous similar messages [ 9430.979479] Lustre: server umount lustre-MDT0000 complete [ 9433.060559] LustreError: 234795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9433.080382] LustreError: 234795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9434.530798] LustreError: 237874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784710 with bad export cookie 13961836329367328523 [ 9434.540418] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9434.554247] LustreError: 237874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9435.037754] Lustre: server umount lustre-MDT0001 complete [ 9448.197916] Lustre: server umount lustre-OST0000 complete [ 9461.396532] Lustre: server umount lustre-OST0001 complete [ 9467.775661] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 9478.899786] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9517.379333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9517.652249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9517.711430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9517.810228] Lustre: lustre-MDT0000: new disk, initializing [ 9517.894338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9517.918883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9522.847546] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9534.217353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9534.362672] Lustre: 241056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9534.433209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9534.437702] Lustre: Skipped 1 previous similar message [ 9534.534727] Lustre: lustre-MDT0001: new disk, initializing [ 9534.642424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9534.662251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9539.325738] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9548.657137] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9554.758593] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9561.401696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9561.655780] Lustre: lustre-OST0000: new disk, initializing [ 9561.660567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9561.670100] Lustre: 243023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9562.988533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9563.008219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9563.153046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9568.509840] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9578.302700] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9583.054654] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9583.237562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9587.996775] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9588.227118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9593.831972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9593.844932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9593.855134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9593.861439] Lustre: Skipped 2 previous similar messages [ 9596.544224] Lustre: server umount lustre-OST0000 complete [ 9601.506424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9601.521256] LustreError: Skipped 1 previous similar message [ 9601.525938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.066103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9604.075549] Lustre: Skipped 3 previous similar messages [ 9615.328895] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9615.566516] Lustre: server umount lustre-MDT0000 complete [ 9618.402750] LustreError: 241062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9618.431979] LustreError: 241062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9619.725864] LustreError: 241049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784895 with bad export cookie 13961836329367330658 [ 9619.732704] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9619.744093] LustreError: 241049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9620.110394] Lustre: server umount lustre-MDT0001 complete [ 9629.049466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9629.534037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9629.539773] Lustre: Skipped 2 previous similar messages [ 9634.151593] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9634.789193] LustreError: 245241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9634.830559] LustreError: 245241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9643.739097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9649.092478] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9656.772348] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9664.186314] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9671.672639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9680.312859] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9689.849326] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9838.051608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9838.068580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9838.072087] Lustre: Skipped 2 previous similar messages [ 9838.074137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9838.099206] Lustre: Skipped 5 previous similar messages [ 9842.038752] Lustre: server umount lustre-OST0000 complete [ 9852.801861] Lustre: server umount lustre-MDT0000 complete [ 9853.409982] LustreError: 247734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.441077] LustreError: 247734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9857.454280] LustreError: 245221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785133 with bad export cookie 13961836329367331540 [ 9857.467157] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9857.479309] LustreError: 245221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9857.850426] Lustre: server umount lustre-MDT0001 complete [ 9958.639541] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9961.566641] Key type lgssc unregistered [ 9961.880309] LNet: 248850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9961.892243] LNetError: 248850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9961.915301] LNet: Removed LNI 192.168.203.151@tcp [ 9962.818202] Key type .llcrypt unregistered [ 9962.820074] Key type ._llcrypt unregistered [ 9988.665251] Key type ._llcrypt registered [ 9988.668485] Key type .llcrypt registered [ 9988.764700] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [10006.629825] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10008.173799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10008.250637] alg: No test for adler32 (adler32-zlib) [10010.014775] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [10010.591714] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [10012.497064] Key type lgssc registered [10014.225106] Lustre: Echo OBD driver; http://www.lustre.org/ [10063.973716] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10074.849395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10074.883269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10076.254522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10076.291401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10076.364973] Lustre: lustre-MDT0000: new disk, initializing [10076.475298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10076.496570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10081.415507] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10093.361062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.467763] Lustre: 252927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10093.518615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10093.521609] Lustre: Skipped 1 previous similar message [10093.602295] Lustre: lustre-MDT0001: new disk, initializing [10093.713958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10093.770309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10093.791577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10098.639696] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10103.474690] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10110.320839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10110.590918] Lustre: lustre-OST0000: new disk, initializing [10110.598664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10110.609613] Lustre: 254557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10110.728994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10116.838722] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10117.769033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10117.786155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10117.945831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10127.536361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10127.669689] Lustre: lustre-OST0001: new disk, initializing [10127.674770] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10127.683925] Lustre: 255428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10127.793089] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10134.584184] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10136.163086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10136.172441] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10136.288677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10147.112907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10156.467363] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10170.339475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10170.349547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.362576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10172.388101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10172.394376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10172.401166] Lustre: Skipped 1 previous similar message [10172.408098] Lustre: Skipped 1 previous similar message [10174.461751] Lustre: server umount lustre-MDT0000 complete [10177.866650] LustreError: 255427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785453 with bad export cookie 6164722021785723592 [10177.895967] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10178.222741] Lustre: server umount lustre-MDT0001 complete [10193.014715] Lustre: server umount lustre-OST0000 complete [10206.977079] Lustre: server umount lustre-OST0001 complete [10222.521475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10233.160479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10241.676916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10252.213666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10271.430613] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10280.817204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10280.847399] Lustre: lustre-MDT0000: reset Object Index mappings [10281.243143] LustreError: 260439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10281.259809] LustreError: 260439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10281.357530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10286.563795] LustreError: 260440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10286.664454] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10291.687844] LustreError: 260439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10295.482875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10295.507609] Lustre: lustre-MDT0001: reset Object Index mappings [10295.922590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10300.432356] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10303.351418] Lustre: 261580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10309.708571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10316.452518] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10320.429940] LustreError: 261934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.211832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10325.625942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10325.631854] Lustre: Skipped 1 previous similar message [10326.664210] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10326.681312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10332.873896] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10341.211884] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10345.373068] Lustre: 263454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10352.097240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10352.111258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10352.127908] Lustre: Skipped 1 previous similar message [10352.135088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.139646] Lustre: Skipped 2 previous similar messages [10352.994575] Lustre: server umount lustre-MDT0000 complete [10356.856529] LustreError: 260419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785632 with bad export cookie 6164722021785730382 [10356.872194] LustreError: 260419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10356.873719] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10357.257766] Lustre: server umount lustre-MDT0001 complete [10371.714215] Lustre: server umount lustre-OST0000 complete [10373.602114] Lustre: 250431:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788785633/real 1788785633] req@ffff979dc26cbb80 x1875677320179584/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788785649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10373.638357] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10373.657268] Lustre: Skipped 3 previous similar messages [10375.701617] Lustre: server umount lustre-OST0001 complete [10394.249299] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 08:54:29 (1788785669) [10411.777860] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [10425.146326] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10464.649341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.806846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10464.859771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10464.993866] Lustre: lustre-MDT0000: new disk, initializing [10465.106552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10465.139386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10470.773243] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10484.673211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10484.916621] Lustre: 267710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10485.047180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10485.063483] Lustre: Skipped 1 previous similar message [10485.232779] Lustre: lustre-MDT0001: new disk, initializing [10485.483465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10485.498539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10491.146441] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10501.251160] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10508.211892] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10515.235499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10515.485393] Lustre: lustre-OST0000: new disk, initializing [10515.490258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10515.504812] Lustre: 269682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10517.571462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10517.585179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10517.754090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10522.388152] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10534.317681] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10539.150505] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10539.366633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10544.854346] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10545.069092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10548.705418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10548.715749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10548.721341] LustreError: Skipped 1 previous similar message [10548.722131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10548.722135] Lustre: Skipped 3 previous similar messages [10548.755163] Lustre: Skipped 1 previous similar message [10553.565407] Lustre: server umount lustre-OST0000 complete [10558.945969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10558.988970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10558.991325] Lustre: Skipped 1 previous similar message [10564.669668] Lustre: server umount lustre-MDT0000 complete [10568.180847] LustreError: 268547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10568.211416] LustreError: 268547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10568.928427] LustreError: 267703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785844 with bad export cookie 6164722021785732475 [10568.930229] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10568.942054] LustreError: 267703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10569.422713] Lustre: server umount lustre-MDT0001 complete [10591.192379] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10601.185900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10601.702023] LustreError: 272615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10601.782450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10601.793444] Lustre: Skipped 2 previous similar messages [10606.130945] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10614.814204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10619.855674] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10622.807304] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10628.906175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10635.342086] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10643.205412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10643.338947] Lustre: lustre-OST0001: new disk, initializing [10643.342314] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10643.347254] Lustre: 274922:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10644.572153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10644.591035] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10644.706448] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10649.587825] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10658.820734] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10662.700319] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10668.227547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10668.252144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10676.714374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10676.732363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10676.746557] Lustre: Skipped 1 previous similar message [10676.753083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10676.756775] Lustre: Skipped 2 previous similar messages [10681.289859] Lustre: server umount lustre-MDT0000 complete [10682.856542] LustreError: 272616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10682.889323] LustreError: 272616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10684.778654] LustreError: 272596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785960 with bad export cookie 6164722021785733357 [10684.790570] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10684.811071] LustreError: 272596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10685.268521] Lustre: server umount lustre-MDT0001 complete [10699.806736] Lustre: server umount lustre-OST0000 complete [10713.354563] Lustre: server umount lustre-OST0001 complete [10734.560561] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10745.472516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10745.896556] LustreError: 277982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10745.962376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10745.979867] Lustre: Skipped 3 previous similar messages [10751.268838] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10760.915133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10766.709368] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10770.736247] Lustre: 279123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10778.533381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10785.608989] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10790.147595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10795.159311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10800.648488] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10802.726709] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10811.572306] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10815.624137] Lustre: 281012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10821.270479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10821.421300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10833.821063] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:01:48 (1788786108) [10851.810314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10851.824901] Lustre: Skipped 5 previous similar messages [10851.833823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10851.843169] Lustre: Skipped 6 previous similar messages [10855.787240] Lustre: server umount lustre-MDT0000 complete [10856.941457] LustreError: 277983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.970519] LustreError: 277983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10860.497290] LustreError: 277964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788786136 with bad export cookie 6164722021785739909 [10860.507609] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10860.510270] LustreError: 277964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10866.940260] Lustre: server umount lustre-MDT0001 complete [10877.372314] Lustre: server umount lustre-OST0000 complete [10881.898299] Lustre: server umount lustre-OST0001 complete [10889.094807] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [10903.731251] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10963.714864] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10975.089798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.411712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10975.451584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10975.657548] Lustre: lustre-MDT0000: new disk, initializing [10975.780306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10981.052416] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10993.172194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10993.292970] Lustre: 285296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10993.651986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10999.141300] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11011.575875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11011.842292] Lustre: 286928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11011.923844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11011.934970] Lustre: Skipped 5 previous similar messages [11013.924949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11018.733360] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11030.156768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11030.247097] Lustre: 287800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11032.068281] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11032.083406] Lustre: Skipped 1 previous similar message [11032.228788] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11036.913833] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11047.931508] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11051.943351] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11051.949965] Lustre: Skipped 1 previous similar message [11071.165703] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:05:45 (1788786345) [11075.553906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11075.559484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.577385] Lustre: Skipped 3 previous similar messages [11075.583408] LustreError: 286096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.600823] LustreError: 286096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11075.650176] Lustre: server umount lustre-MDT0000 complete [11079.250320] LustreError: 286090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788786355 with bad export cookie 6164722021785747553 [11079.260475] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11079.266548] LustreError: 286090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11079.681085] Lustre: server umount lustre-MDT0001 complete [11093.956883] Lustre: server umount lustre-OST0000 complete [11107.760574] Lustre: server umount lustre-OST0001 complete [11117.456465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11122.499347] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11131.300779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11135.933090] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11143.942083] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11150.832277] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11158.954678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11166.772199] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11174.993191] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11195.812289] loop1: detected capacity change from 0 to 204800000 [11196.107864] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11196.128611] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11196.162952] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11202.194554] loop1: detected capacity change from 0 to 204800000 [11208.252707] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11208.373384] LustreError: 290037:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs11594: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11208.390898] LustreError: 290037:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs11594 [11208.396917] LustreError: 290037:0:(mgs_handler.c:517:mgs_target_reg()) fs11594-MDT0001: Error checking target index: rc = -13 [11208.419251] LustreError: fs11594-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11208.441329] LustreError: 293771:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11208.451455] LustreError: 293771:0:(tgt_mount.c:2132:server_put_super()) no obd fs11594-MDT0001 [11208.464910] LustreError: 293771:0:(tgt_mount.c:116:server_deregister_mount()) fs11594-MDT0001 not registered [11208.539684] Lustre: server umount fs11594-MDT0001 complete [11208.550133] LustreError: 293771:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11218.903569] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11219.114475] Lustre: Setting parameter fs11594-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs11594-MDT0001 [11219.164995] Lustre: srv-fs11594-MDT0001: No data found on store. Initialize space. [11219.181399] Lustre: Skipped 4 previous similar messages [11219.346463] Lustre: fs11594-MDT0001: new disk, initializing [11219.354598] Lustre: Skipped 3 previous similar messages [11219.562454] Lustre: srv-fs11594-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11219.579617] Lustre: 294237:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11594-MDT0001-mdtlov: can't create new llog: rc = -115 [11219.600694] LustreError: 294237:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11594-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11224.920031] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11231.814855] Lustre: server umount fs11594-MDT0001 complete [11242.985453] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11245.417767] loop1: detected capacity change from 0 to 204800000 [11251.614544] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.282748] Lustre: 295806:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11594-MDT0001-mdtlov: can't create new llog: rc = -115 [11252.298996] LustreError: 295806:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11594-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11257.138126] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11262.470092] Lustre: server umount fs11594-MDT0001 complete [11273.391123] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [11273.410583] Lustre: Skipped 4 previous similar messages [11273.707543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11273.718251] Lustre: Skipped 5 previous similar messages [11278.375583] Lustre: server umount lustre-MDT0000 complete [11287.875572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.018909] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11293.284863] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11293.700986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11303.074597] loop1: detected capacity change from 0 to 204800000 [11309.698384] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.452925] Lustre: 298254:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11594-MDT0001-mdtlov: can't create new llog: rc = -115 [11310.476434] LustreError: 298254:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11594-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11315.416718] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11320.539388] Lustre: server umount fs11594-MDT0001 complete [11334.640520] LustreError: 291881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11334.663406] LustreError: 291881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 109 previous similar messages [11345.378097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11345.396361] LustreError: Skipped 1 previous similar message [11345.414215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11345.422815] Lustre: Skipped 3 previous similar messages [11359.200138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11359.405427] Lustre: server umount lustre-OST0000 complete [11369.545464] Lustre: server umount lustre-MDT0000 complete [11373.315930] LustreError: 290030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788786649 with bad export cookie 6164722021785750241 [11373.321281] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11373.334368] LustreError: 290030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11373.688554] Lustre: server umount lustre-MDT0001 complete [11386.411278] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11389.004298] Key type lgssc unregistered [11389.285732] LNet: 300599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11389.295967] LNetError: 300599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11389.315972] LNet: Removed LNI 192.168.203.151@tcp [11390.153663] Key type .llcrypt unregistered [11390.157988] Key type ._llcrypt unregistered [11415.292467] Key type ._llcrypt registered [11415.294752] Key type .llcrypt registered [11415.512095] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11416.891842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11416.989455] alg: No test for adler32 (adler32-zlib) [11418.176018] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [11418.394422] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11420.096172] Key type lgssc registered [11421.408415] Lustre: Echo OBD driver; http://www.lustre.org/ [11429.692414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11429.723847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11431.253759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11437.101853] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11441.635259] LustreError: 301867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11441.664809] LustreError: 301867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11446.753762] LustreError: 301866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11446.997219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.543783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11452.395912] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11461.581765] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11469.282838] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11477.805725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11478.834239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11486.865310] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11492.147242] LustreError: 303699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11497.174367] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11497.455758] LustreError: 303700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11501.549361] LustreError: 303699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11501.579535] LustreError: 303699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11511.798160] LustreError: 303699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11511.820135] LustreError: 303699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11520.158880] loop1: detected capacity change from 0 to 204800000 [11520.406668] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11520.423781] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11520.435793] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11525.609238] loop1: detected capacity change from 0 to 204800000 [11528.830329] LustreError: 303699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11528.852775] LustreError: 303699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11531.791295] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11531.921326] Lustre: fs22119-OST0000: new disk, initializing [11531.926059] Lustre: srv-fs22119-OST0000: No data found on store. Initialize space. [11531.939021] Lustre: 305606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11532.016875] Lustre: fs22119-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11538.562519] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11547.262273] Lustre: server umount fs22119-OST0000 complete [11560.231917] loop1: detected capacity change from 0 to 204800000 [11562.987115] LustreError: 303700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11563.004796] LustreError: 303700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11567.311787] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11567.626759] Lustre: fs22119-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11573.226304] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11575.443875] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11578.343091] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11587.266867] loop2: detected capacity change from 0 to 204800000 [11587.524921] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11587.541334] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11587.557819] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11593.015491] loop2: detected capacity change from 0 to 204800000 [11598.764696] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11598.854658] LustreError: fs22119-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11598.879609] LustreError: fs22119-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11598.885377] LustreError: 308647:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11598.890892] LustreError: 308647:0:(tgt_mount.c:2132:server_put_super()) no obd fs22119-OST0001 [11598.899170] LustreError: 308647:0:(tgt_mount.c:116:server_deregister_mount()) fs22119-OST0001 not registered [11598.947537] Lustre: server umount fs22119-OST0001 complete [11598.949667] LustreError: 308647:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11610.657597] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11610.800868] Lustre: fs22119-OST0001: new disk, initializing [11610.812915] Lustre: srv-fs22119-OST0001: No data found on store. Initialize space. [11610.826126] Lustre: 309199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11610.925894] Lustre: fs22119-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11618.095364] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11628.432878] Lustre: server umount fs22119-OST0000 complete [11629.551090] LustreError: 303698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11629.599480] LustreError: 303698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11640.113903] Lustre: server umount fs22119-OST0001 complete [11653.686835] loop1: detected capacity change from 0 to 204800000 [11660.041753] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11660.288962] Lustre: fs22119-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11667.476309] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11675.038122] loop2: detected capacity change from 0 to 204800000 [11681.623983] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11690.498789] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11701.758523] Lustre: server umount fs22119-OST0000 complete [11713.294467] Lustre: server umount fs22119-OST0001 complete [11726.836977] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11731.943755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11731.943906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11731.950880] LustreError: Skipped 1 previous similar message [11731.951981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11731.976366] Lustre: Skipped 1 previous similar message [11735.123745] Lustre: server umount lustre-OST0000 complete [11739.618134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11739.622422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11739.632209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11739.635750] Lustre: Skipped 1 previous similar message [11742.179187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11742.181765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11742.232426] Lustre: Skipped 1 previous similar message [11745.361592] Lustre: server umount lustre-MDT0000 complete [11749.545959] LustreError: 301848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787025 with bad export cookie 3546864448622922944 [11749.558682] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11749.573887] LustreError: 301848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11750.118853] Lustre: server umount lustre-MDT0001 complete [11764.188876] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11767.863308] Key type lgssc unregistered [11768.244870] LNet: 315121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11768.255193] LNetError: 315121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11768.275783] LNet: Removed LNI 192.168.203.151@tcp [11769.434165] Key type .llcrypt unregistered [11769.439574] Key type ._llcrypt unregistered [11810.955257] Key type ._llcrypt registered [11810.957939] Key type .llcrypt registered [11811.111167] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [11832.397148] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11833.459277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11833.575776] alg: No test for adler32 (adler32-zlib) [11834.870657] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [11835.160909] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11836.840206] Key type lgssc registered [11837.873096] Lustre: Echo OBD driver; http://www.lustre.org/ [11878.710196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11878.825727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11880.215877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11880.257161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11880.397125] Lustre: lustre-MDT0000: new disk, initializing [11880.516941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11880.546203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11885.722527] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11898.435699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11898.579541] Lustre: 318977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11898.638376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11898.655316] Lustre: Skipped 1 previous similar message [11898.821127] Lustre: lustre-MDT0001: new disk, initializing [11898.976307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11899.075568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11899.102921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11904.411691] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11916.163384] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11924.773512] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11932.507534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11932.851621] Lustre: lustre-OST0000: new disk, initializing [11932.855952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11932.865507] Lustre: 320948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11932.943912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11939.918711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11939.937786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11940.035601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11940.317504] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11950.163462] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11952.363248] Lustre: Mounted lustre-client - version 2.17.57_103_g80273de [11953.402872] Lustre: Unmounted lustre-client [11965.921688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11965.922561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11965.941603] LustreError: Skipped 1 previous similar message [11965.943608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11965.987531] Lustre: Skipped 1 previous similar message [11967.991211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11968.002694] Lustre: Skipped 2 previous similar messages [11969.508487] Lustre: server umount lustre-OST0000 complete [11975.649509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11975.657923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11975.678319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11978.210071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11978.211449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11979.961834] Lustre: server umount lustre-MDT0000 complete [11983.944826] LustreError: 318971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787259 with bad export cookie 17298007590002446591 [11983.945382] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11983.950764] LustreError: 318971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11984.380283] Lustre: server umount lustre-MDT0001 complete [11997.216386] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12000.639454] Key type lgssc unregistered [12004.130281] LNet: 1 local NIs in recovery (showing 1): 192.168.203.151@tcp [12005.091789] LNet: 323111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12005.111283] LNetError: 323111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12085.038876] LNet: Removed LNI 192.168.203.151@tcp [12086.221854] Key type .llcrypt unregistered [12086.224972] Key type ._llcrypt unregistered [12109.751349] Key type ._llcrypt registered [12109.753662] Key type .llcrypt registered [12109.865321] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [12128.817212] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12130.407618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12130.477388] alg: No test for adler32 (adler32-zlib) [12131.770867] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [12131.964656] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12133.616170] Key type lgssc registered [12135.031624] Lustre: Echo OBD driver; http://www.lustre.org/ [12174.221804] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:24:07 (1788787447) [12184.239416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12184.282934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12185.837563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12185.908588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12186.026799] Lustre: lustre-MDT0000: new disk, initializing [12186.088538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12186.105052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12192.106661] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12206.531835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.706542] Lustre: 326333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12206.783533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12206.788109] Lustre: Skipped 1 previous similar message [12206.880956] Lustre: lustre-MDT0001: new disk, initializing [12207.008802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12207.067431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12207.083319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12212.458150] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12223.393423] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12230.266277] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12238.713490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12239.033974] Lustre: lustre-OST0000: new disk, initializing [12239.037731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12239.051059] Lustre: 328301:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12239.150860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12246.438629] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12248.178065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12248.196985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12248.295086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12257.374630] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12279.291894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12279.302944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12279.308610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12279.341650] Lustre: Skipped 1 previous similar message [12282.447107] Lustre: server umount lustre-OST0000 complete [12288.994789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12289.000017] LustreError: Skipped 1 previous similar message [12289.002609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12289.013223] Lustre: Skipped 1 previous similar message [12289.017088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12294.321678] Lustre: server umount lustre-MDT0000 complete [12299.332401] LustreError: 326326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787575 with bad export cookie 16062592189140589071 [12299.348812] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12299.379528] LustreError: 326326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12299.989736] Lustre: server umount lustre-MDT0001 complete [12313.508372] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12316.827834] Key type lgssc unregistered [12317.082613] LNet: 330211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12317.091060] LNetError: 330211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12317.116135] LNet: Removed LNI 192.168.203.151@tcp [12318.267690] Key type .llcrypt unregistered [12318.273318] Key type ._llcrypt unregistered [12346.664533] Key type ._llcrypt registered [12346.669473] Key type .llcrypt registered [12346.842767] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12348.309507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12348.325703] alg: No test for adler32 (adler32-zlib) [12349.806337] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [12350.139297] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12352.048343] Key type lgssc registered [12353.080664] Lustre: Echo OBD driver; http://www.lustre.org/ [12362.008482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12362.018205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12363.349730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12368.437631] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12373.499427] LustreError: 331489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12373.524900] LustreError: 331489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12378.367092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12378.599478] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12378.913981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12384.712900] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12393.648881] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12400.067709] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12407.668672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12408.187107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12415.009534] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12423.692913] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12435.248991] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12437.252624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12444.132157] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12444.152498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12444.175163] Lustre: Skipped 2 previous similar messages [12444.195966] Lustre: Skipped 2 previous similar messages [12446.869303] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [12448.872368] Lustre: server umount lustre-MDT0000 complete [12449.264959] LustreError: 331484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12449.294097] LustreError: 331484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12451.952891] LustreError: 331485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12453.676229] LustreError: 332222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787729 with bad export cookie 8835134465843789060 [12453.678236] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12453.685491] LustreError: 332222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12454.370965] LustreError: 332926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12454.374259] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12454.401208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12458.401454] LustreError: 331484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12458.402581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12458.438709] LustreError: 331484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12458.460458] Lustre: Skipped 1 previous similar message [12460.309528] Lustre: server umount lustre-MDT0001 complete [12464.851143] Lustre: server umount lustre-OST0000 complete [12474.316071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12479.930885] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12483.559133] Lustre: 335675:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.151@tcp [12485.824890] Lustre: server umount MGS complete [12495.454129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12495.992169] LustreError: 336246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12496.110340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12500.826568] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12509.240488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12509.620649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12514.272554] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12522.421565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12522.745725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12528.667833] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12536.028391] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12537.996652] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12545.007411] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12547.795302] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [12554.721895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12554.731053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12554.750343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12557.801499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12557.817608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12560.441734] Lustre: server umount lustre-OST0000 complete [12565.986434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12565.995946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12570.431175] Lustre: server umount lustre-MDT0000 complete [12573.159826] LustreError: 336247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12573.174077] LustreError: 336247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [12573.936710] LustreError: 336228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787849 with bad export cookie 8835134465843789900 [12573.940502] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12573.950486] LustreError: 336228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12574.346205] Lustre: server umount lustre-MDT0001 complete [12585.697607] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12588.495315] Key type lgssc unregistered [12588.783709] LNet: 339681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12588.797467] LNetError: 339681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12589.875530] LNet: Removed LNI 192.168.203.151@tcp [12590.834826] Key type .llcrypt unregistered [12590.837688] Key type ._llcrypt unregistered [12609.105806] Key type ._llcrypt registered [12609.108504] Key type .llcrypt registered [12609.236541] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [12625.083547] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12625.989804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12626.076075] alg: No test for adler32 (adler32-zlib) [12627.133247] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [12627.373308] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12629.032181] Key type lgssc registered [12630.181929] Lustre: Echo OBD driver; http://www.lustre.org/ [12665.803245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12665.856161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12667.204376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12667.248878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12667.382040] Lustre: lustre-MDT0000: new disk, initializing [12667.531356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12667.565486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12672.048598] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12682.863899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12682.991369] Lustre: 342848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12683.059815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12683.064433] Lustre: Skipped 1 previous similar message [12683.142280] Lustre: lustre-MDT0001: new disk, initializing [12683.257801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12683.303479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12683.324738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12687.474538] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12696.301683] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12702.335646] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12709.450417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12709.918176] Lustre: lustre-OST0000: new disk, initializing [12709.929514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12709.944600] Lustre: 344815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12710.129662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12717.744502] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12720.281503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12720.287533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12720.407397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12730.919571] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12737.255204] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12737.509414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12742.459817] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12742.710139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12746.222608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12746.233487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12746.235520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12746.241832] Lustre: Skipped 1 previous similar message [12746.271855] LustreError: Skipped 1 previous similar message [12751.145648] Lustre: server umount lustre-OST0000 complete [12756.449731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12756.452310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12756.459731] Lustre: Skipped 1 previous similar message [12756.470155] Lustre: Skipped 2 previous similar messages [12761.575928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12761.586273] Lustre: Skipped 1 previous similar message [12762.143229] Lustre: server umount lustre-MDT0000 complete [12766.155435] LustreError: 343642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788788042 with bad export cookie 10074083697350065990 [12766.158575] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12766.161962] LustreError: 343642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12766.483211] Lustre: server umount lustre-MDT0001 complete [12775.918342] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:34:11 (1788788051) [12792.260147] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [12804.622745] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12845.484921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12845.872703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12845.943530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12846.162734] Lustre: lustre-MDT0000: new disk, initializing [12846.331588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12846.350825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12851.432649] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12862.942175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12863.120518] Lustre: 349576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12863.224405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12863.233341] Lustre: Skipped 1 previous similar message [12863.343420] Lustre: lustre-MDT0001: new disk, initializing [12863.490545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12863.562413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12863.575305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12868.299606] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12878.041276] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12884.362410] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12890.771203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12891.012331] Lustre: lustre-OST0000: new disk, initializing [12891.016471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12891.023668] Lustre: 351543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12891.116641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12897.668448] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12900.934097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12900.942400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12901.076521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12907.492607] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12947.427039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12947.429844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12947.444185] LustreError: Skipped 1 previous similar message [12947.475326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12950.388985] Lustre: server umount lustre-OST0000 complete [12955.618502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12955.634759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12955.654753] Lustre: Skipped 1 previous similar message [12955.660951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12955.673418] Lustre: Skipped 1 previous similar message [12960.828501] Lustre: server umount lustre-MDT0000 complete [12965.199783] LustreError: 349568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788788241 with bad export cookie 10074083697350066872 [12965.216601] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12965.221629] LustreError: 349568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12965.622955] Lustre: server umount lustre-MDT0001 complete [12978.408892] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12981.211909] Key type lgssc unregistered [12981.493867] LNet: 353719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12981.500888] LNetError: 353719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12982.583970] LNet: Removed LNI 192.168.203.151@tcp [12983.647405] Key type .llcrypt unregistered [12983.651245] Key type ._llcrypt unregistered [13083.272194] Key type ._llcrypt registered [13083.285479] Key type .llcrypt registered [13083.457564] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13083.832391] Key type .llcrypt unregistered [13083.839603] Key type ._llcrypt unregistered [13163.489727] Key type ._llcrypt registered [13163.492502] Key type .llcrypt registered [13163.652515] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13164.056429] Key type .llcrypt unregistered [13164.060488] Key type ._llcrypt unregistered [13195.215307] Key type ._llcrypt registered [13195.218824] Key type .llcrypt registered [13195.391963] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13195.673903] Key type .llcrypt unregistered [13195.681421] Key type ._llcrypt unregistered [13229.055313] Key type ._llcrypt registered [13229.057408] Key type .llcrypt registered [13229.222330] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13229.592453] Key type .llcrypt unregistered [13229.597352] Key type ._llcrypt unregistered [13322.537560] Key type ._llcrypt registered [13322.542273] Key type .llcrypt registered [13322.700451] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13323.021336] Key type .llcrypt unregistered [13323.024986] Key type ._llcrypt unregistered [13354.104149] Key type ._llcrypt registered [13354.105903] Key type .llcrypt registered [13354.189702] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13354.442658] Key type .llcrypt unregistered [13354.448310] Key type ._llcrypt unregistered [13389.350353] Key type ._llcrypt registered [13389.354607] Key type .llcrypt registered [13389.476830] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13390.070279] Key type .llcrypt unregistered [13390.076878] Key type ._llcrypt unregistered [13424.261791] Key type ._llcrypt registered [13424.264468] Key type .llcrypt registered [13424.398509] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13424.604540] Key type .llcrypt unregistered [13424.606452] Key type ._llcrypt unregistered [13454.594148] Key type ._llcrypt registered [13454.597119] Key type .llcrypt registered [13454.726888] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13455.066605] Key type .llcrypt unregistered [13455.073263] Key type ._llcrypt unregistered [13485.951291] Key type ._llcrypt registered [13485.956630] Key type .llcrypt registered [13486.084533] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13486.366849] Key type .llcrypt unregistered [13486.371247] Key type ._llcrypt unregistered [13552.153399] Key type ._llcrypt registered [13552.160152] Key type .llcrypt registered [13552.296178] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13552.653442] Key type .llcrypt unregistered [13552.661529] Key type ._llcrypt unregistered [13585.584424] Key type ._llcrypt registered [13585.591608] Key type .llcrypt registered [13585.730797] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13585.946310] Key type .llcrypt unregistered [13585.953971] Key type ._llcrypt unregistered [13620.931903] Key type ._llcrypt registered [13620.944843] Key type .llcrypt registered [13621.060855] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13621.431801] Key type .llcrypt unregistered [13621.435122] Key type ._llcrypt unregistered [13686.711525] Key type ._llcrypt registered [13686.714534] Key type .llcrypt registered [13686.850102] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [13704.000642] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [13705.010171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13705.038117] alg: No test for adler32 (adler32-zlib) [13706.316716] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [13706.570379] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [13708.280564] Key type lgssc registered [13709.787781] Lustre: Echo OBD driver; http://www.lustre.org/ [13753.782227] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13753.842799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13755.055474] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13755.121203] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13755.217365] Lustre: tst250fs-MDT0000: new disk, initializing [13755.352675] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13755.374189] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13760.134521] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13771.515264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13771.628936] Lustre: 374029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13771.692232] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13771.699452] Lustre: Skipped 1 previous similar message [13771.801898] Lustre: tst250fs-MDT0001: new disk, initializing [13771.952334] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13772.005310] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13772.016795] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13777.523122] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13782.220867] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13790.147625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13790.505923] Lustre: tst250fs-OST0000: new disk, initializing [13790.514354] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13790.523539] Lustre: 375659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13790.642917] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13796.940378] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13796.956416] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13797.087286] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13797.360355] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13809.130830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13809.344142] Lustre: tst250fs-OST0001: new disk, initializing [13809.349926] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13809.356828] Lustre: 376532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13809.477194] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13815.887288] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13819.499604] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13819.508277] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13819.607564] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13821.874879] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13824.403836] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13824.409273] Lustre: Skipped 2 previous similar messages [13843.939208] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13843.957784] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13843.981359] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13845.476741] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13845.481709] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13845.506254] Lustre: Skipped 2 previous similar messages [13845.527109] Lustre: Skipped 2 previous similar messages [13849.445698] Lustre: server umount tst250fs-MDT0000 complete [13853.013565] LustreError: 374020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788789129 with bad export cookie 4789695333888193722 [13853.019214] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13853.022166] LustreError: 374020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13853.476392] Lustre: server umount tst250fs-MDT0001 complete [13867.346202] Lustre: server umount tst250fs-OST0000 complete [13871.073161] Lustre: 372249:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788789131/real 1788789131] req@ffff979dc1c4ad80 x1875681195657216/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788789147 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13871.116533] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13872.468642] Lustre: server umount tst250fs-OST0001 complete [13895.241600] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [13903.805732] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [13940.140082] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 09:53:35 (1788789215) [13942.152096] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13958.263653] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [13968.834663] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14003.285766] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13740 sec ========= 09:54:38 (1788789278) [14005.251581] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:54:40 (1788789280) === [14009.019296] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:54:44 (1788789284) === [14049.471868] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [14052.517507] Key type lgssc unregistered [14052.745869] LNet: 385658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14052.769700] LNetError: 385658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14052.808710] LNet: Removed LNI 192.168.203.151@tcp [14053.617207] Key type .llcrypt unregistered [14053.622304] Key type ._llcrypt unregistered