[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505159721 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002335] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006911] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009122] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013054] *** VALIDATE selinux *** [ 0.020726] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024590] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025127] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027099] *** VALIDATE tmpfs *** [ 0.029315] *** VALIDATE proc *** [ 0.030218] *** VALIDATE cgroup *** [ 0.031010] *** VALIDATE cgroup2 *** [ 0.033231] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036026] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.039858] debug: unmapping init [mem 0xffffffffaf059000-0xffffffffaf060fff] [ 0.041152] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042617] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043018] ... version: 2 [ 0.044010] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.046011] ... value mask: 0000ffffffffffff [ 0.047012] ... max period: 00007fffffffffff [ 0.048024] ... fixed-purpose events: 3 [ 0.049014] ... event mask: 000000070000000f [ 0.050263] rcu: Hierarchical SRCU implementation. [ 0.052363] smp: Bringing up secondary CPUs ... [ 0.053508] x86: Booting SMP configuration: [ 0.054020] .... node #0, CPUs: #1 #2 #3 [ 0.057534] smp: Brought up 1 node, 4 CPUs [ 0.059011] smpboot: Max logical packages: 1 [ 0.060015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138312] node 0 deferred pages initialised in 76ms [ 0.142093] devtmpfs: initialized [ 0.144363] x86/mm: Memory block size: 128MB [ 0.148123] gcov: version magic: 0x41383552 [ 0.152316] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154254] pinctrl core: initialized pinctrl subsystem [ 0.155251] [ 0.155798] ************************************************************* [ 0.157014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160014] ** ** [ 0.162012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165014] ** ** [ 0.167008] ** This means that this kernel is built to expose internal ** [ 0.169012] ** IOMMU data structures, which may compromise security on ** [ 0.171014] ** your system. ** [ 0.174014] ** ** [ 0.177013] ** If you see this message and you are not debugging the ** [ 0.179011] ** kernel, report this immediately to your vendor! ** [ 0.182013] ** ** [ 0.184010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187020] ************************************************************* [ 0.189664] NET: Registered protocol family 16 [ 0.192519] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207104] cpuidle: using governor menu [ 0.210101] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213502] PCI: Using configuration type 1 for base access [ 0.216118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226075] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.232038] cryptd: max_cpu_qlen set to 1000 [ 0.234298] ACPI: Added _OSI(Module Device) [ 0.235016] ACPI: Added _OSI(Processor Device) [ 0.237012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244296] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251696] ACPI: Interpreter enabled [ 0.253049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255015] ACPI: Using IOAPIC for interrupt routing [ 0.257121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.261370] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271633] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.274040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.279080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284162] acpiphp: Slot [2] registered [ 0.285062] acpiphp: Slot [5] registered [ 0.286091] acpiphp: Slot [6] registered [ 0.287082] acpiphp: Slot [7] registered [ 0.289094] acpiphp: Slot [8] registered [ 0.290074] acpiphp: Slot [9] registered [ 0.291096] acpiphp: Slot [10] registered [ 0.292079] acpiphp: Slot [3] registered [ 0.293040] acpiphp: Slot [4] registered [ 0.294034] acpiphp: Slot [11] registered [ 0.295047] acpiphp: Slot [12] registered [ 0.296086] acpiphp: Slot [13] registered [ 0.298088] acpiphp: Slot [14] registered [ 0.299065] acpiphp: Slot [15] registered [ 0.300109] acpiphp: Slot [16] registered [ 0.302127] acpiphp: Slot [17] registered [ 0.303156] acpiphp: Slot [18] registered [ 0.306132] acpiphp: Slot [19] registered [ 0.307090] acpiphp: Slot [20] registered [ 0.308097] acpiphp: Slot [21] registered [ 0.310112] acpiphp: Slot [22] registered [ 0.312097] acpiphp: Slot [23] registered [ 0.313093] acpiphp: Slot [24] registered [ 0.315133] acpiphp: Slot [25] registered [ 0.317111] acpiphp: Slot [26] registered [ 0.318094] acpiphp: Slot [27] registered [ 0.320107] acpiphp: Slot [28] registered [ 0.321099] acpiphp: Slot [29] registered [ 0.323086] acpiphp: Slot [30] registered [ 0.324088] acpiphp: Slot [31] registered [ 0.326065] PCI host bridge to bus 0000:00 [ 0.328029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.330024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.332022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.334030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.336084] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341235] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344010] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348352] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357712] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.363008] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.364015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.368013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.369485] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373721] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375028] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377674] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.396016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.400996] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.407706] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.432013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.457035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.469426] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.479016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.486014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.518571] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.528017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.536013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.553016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.565911] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.579029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.588015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.607018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.619794] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.629014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.638018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.653052] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.663341] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.681016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.698015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.708130] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.710443] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.714374] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.716332] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.718207] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.723069] iommu: Default domain type: Passthrough [ 0.724000] SCSI subsystem initialized [ 0.725132] ACPI: bus type USB registered [ 0.728111] usbcore: registered new interface driver usbfs [ 0.730115] usbcore: registered new interface driver hub [ 0.732091] usbcore: registered new device driver usb [ 0.734173] pps_core: LinuxPPS API ver. 1 registered [ 0.736015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.740069] PTP clock support registered [ 0.742072] EDAC MC: Ver: 3.0.0 [ 0.743566] PCI: Using ACPI for IRQ routing [ 0.745848] NetLabel: Initializing [ 0.747009] NetLabel: domain hash size = 128 [ 0.749014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.751111] NetLabel: unlabeled traffic allowed by default [ 0.754088] vgaarb: loaded [ 0.755315] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.757012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.762781] clocksource: Switched to clocksource kvm-clock [ 0.870045] VFS: Disk quotas dquot_6.6.0 [ 0.871731] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.874356] *** VALIDATE ramfs *** [ 0.875646] *** VALIDATE hugetlbfs *** [ 0.877063] pnp: PnP ACPI init [ 0.879492] pnp: PnP ACPI: found 6 devices [ 0.897310] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.900114] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.902771] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.905427] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.908243] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.910825] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.913744] NET: Registered protocol family 2 [ 0.916285] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.921237] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.924819] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.930304] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.933515] TCP: Hash tables configured (established 65536 bind 65536) [ 0.936374] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.939599] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.942430] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.945435] NET: Registered protocol family 1 [ 0.948184] RPC: Registered named UNIX socket transport module. [ 0.950137] RPC: Registered udp transport module. [ 0.951852] RPC: Registered tcp transport module. [ 0.953394] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.954977] NET: Registered protocol family 44 [ 0.956218] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.957934] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.959450] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.961242] PCI: CLS 0 bytes, default 64 [ 0.962667] Unpacking initramfs... [ 2.346487] debug: unmapping init [mem 0xffff92913cc54000-0xffff92913ffbffff] [ 2.350151] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.352455] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.356292] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.857167] Initialise system trusted keyrings [ 2.858869] Key type blacklist registered [ 2.860726] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.872891] zbud: loaded [ 2.876431] *** VALIDATE nfs *** [ 2.877744] *** VALIDATE nfs4 *** [ 2.879454] pstore: using deflate compression [ 2.883138] Platform Keyring initialized [ 2.989532] NET: Registered protocol family 38 [ 2.991291] Key type asymmetric registered [ 2.992839] Asymmetric key parser 'x509' registered [ 2.994841] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.006682] io scheduler mq-deadline registered [ 3.009423] io scheduler kyber registered [ 3.011081] io scheduler bfq registered [ 3.013084] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.015695] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.020734] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.024276] ACPI: Power Button [PWRF] [ 3.033046] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.041284] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.052831] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.058607] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.075228] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.105536] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.136450] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.141051] Non-volatile memory driver v1.3 [ 3.142750] Linux agpgart interface v0.103 [ 3.175886] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.178640] vda: detected capacity change from 0 to 76677120 [ 3.194850] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.198075] vdb: detected capacity change from 0 to 1073741824 [ 3.215119] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.218094] vdc: detected capacity change from 0 to 2621440000 [ 3.242739] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.246493] vdd: detected capacity change from 0 to 2621440000 [ 3.261275] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.264915] vde: detected capacity change from 0 to 4294967296 [ 3.281196] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.283890] vdf: detected capacity change from 0 to 4294967296 [ 3.290876] libphy: Fixed MDIO Bus: probed [ 3.296513] usbcore: registered new interface driver usbserial_generic [ 3.299559] usbserial: USB Serial support registered for generic [ 3.302170] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.306435] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.310207] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.312746] mousedev: PS/2 mouse device common for all mice [ 3.315280] rtc_cmos 00:05: RTC can wake from S4 [ 3.318109] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.319279] rtc_cmos 00:05: registered as rtc0 [ 3.324422] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.325281] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.331681] intel_pstate: CPU model not supported [ 3.331756] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.338235] hid: raw HID events driver (C) Jiri Kosina [ 3.340731] usbcore: registered new interface driver usbhid [ 3.342867] usbhid: USB HID core driver [ 3.344613] drop_monitor: Initializing network drop monitor service [ 3.347258] Initializing XFRM netlink socket [ 3.349566] NET: Registered protocol family 10 [ 3.353277] Segment Routing with IPv6 [ 3.354605] NET: Registered protocol family 17 [ 3.356842] mpls_gso: MPLS GSO support [ 3.363432] RAS: Correctable Errors collector initialized. [ 3.365770] AVX version of gcm_enc/dec engaged. [ 3.367385] AES CTR mode by8 optimization enabled [ 3.447695] sched_clock: Marking stable (3447672626, 0)->(4371879516, -924206890) [ 3.451275] registered taskstats version 1 [ 3.453813] Loading compiled-in X.509 certificates [ 3.456306] zswap: loaded using pool lzo/zbud [ 3.480667] Key type big_key registered [ 3.494931] Key type encrypted registered [ 3.496880] ima: No TPM chip found, activating TPM-bypass! [ 3.498861] ima: Allocated hash algorithm: sha1 [ 3.500358] ima: No architecture policies found [ 3.502087] evm: Initialising EVM extended attributes: [ 3.504074] evm: security.selinux [ 3.505380] evm: security.ima [ 3.506637] evm: security.capability [ 3.508075] evm: HMAC attrs: 0x1 [ 3.510428] rtc_cmos 00:05: setting system clock to 2026-09-03 12:20:34 UTC (1788438034) [ 3.516879] debug: unmapping init [mem 0xffffffffb0003000-0xffffffffb01fffff] [ 3.519890] debug: unmapping init [mem 0xffffffffaed82000-0xffffffffaf058fff] [ 3.528075] Write protecting the kernel read-only data: 28672k [ 3.531739] debug: unmapping init [mem 0xffffffffad403000-0xffffffffad5fffff] [ 3.535444] debug: unmapping init [mem 0xffffffffadd14000-0xffffffffaddfffff] [ 3.571283] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.579644] systemd[1]: Detected virtualization kvm. [ 3.581230] systemd[1]: Detected architecture x86-64. [ 3.583341] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.609353] systemd[1]: No hostname configured. [ 3.611196] systemd[1]: Set hostname to . [ 3.613194] random: systemd: uninitialized urandom read (16 bytes read) [ 3.616234] systemd[1]: Initializing machine ID from random generator. [ 3.750163] random: systemd: uninitialized urandom read (16 bytes read) [ 3.753432] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.758708] random: systemd: uninitialized urandom read (16 bytes read) [ 3.763176] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.768863] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.485542] device-mapper: uevent: version 1.0.3 [ 4.487644] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.309547] random: fast init done [ 5.323524] virtio_net virtio0 ens2: renamed from eth0 [ 5.401258] scsi host0: ata_piix [ 5.451163] scsi host1: ata_piix [ 5.452472] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.454217] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.235683] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.014750] random: crng init done [ 10.016035] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.646191] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.841483] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.148429] SELinux: Disabled at runtime. [ 12.212516] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.224420] systemd[1]: Detected virtualization kvm. [ 12.226385] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.800078] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.804977] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.811506] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.815932] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.819959] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.829299] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.834826] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ 13.052566] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.350486] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.694977] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.760098] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.903140] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.922436] EDAC sbridge: Ver: 1.1.2 [ 15.770272] Key type dns_resolver registered [ 16.082497] NFS: Registering the id_resolver key type [ 16.084691] Key type id_resolver registered [ 16.086363] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg657-server login: [ 42.262293] libcfs: loading out-of-tree module taints kernel. [ 42.283209] Key type ._llcrypt registered [ 42.284895] Key type .llcrypt registered [ 42.338270] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 59.320577] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 61.068903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.102207] alg: No test for adler32 (adler32-zlib) [ 62.448331] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 63.381340] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 65.175250] Key type lgssc registered [ 66.934291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.801484] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 126.103708] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 129.126059] hrtimer: interrupt took 6048555 ns [ 138.341631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 138.394717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.590974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 139.625763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.773275] Lustre: lustre-MDT0000: new disk, initializing [ 139.852948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.876763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.794532] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.881444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.975917] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 157.054522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 157.062966] Lustre: Skipped 1 previous similar message [ 157.160941] Lustre: lustre-MDT0001: new disk, initializing [ 157.223903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.276743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.302246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 161.278250] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.317842] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 176.607593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 177.090115] Lustre: lustre-OST0000: new disk, initializing [ 177.094453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 177.108568] Lustre: 8460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 177.197689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.252519] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.886079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.895280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.030162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.373202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.494222] Lustre: lustre-OST0001: new disk, initializing [ 198.497912] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 198.504422] Lustre: 9533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.567082] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.795813] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.436373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 206.449641] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 206.539155] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 217.498948] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.110728] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 231.753679] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing check_logdir /tmp/testlogs/ [ 236.622921] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing yml_node [ 241.131046] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 243.757765] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 246.501651] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 248.355774] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:24:37 EDT 2026 [ 265.617392] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 267.143681] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 283.620249] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.621345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 283.646029] Lustre: Skipped 2 previous similar messages [ 288.736486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.750181] Lustre: Skipped 6 previous similar messages [ 289.937592] Lustre: server umount lustre-MDT0000 complete [ 293.206487] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438324 with bad export cookie 11874256787795256877 [ 293.207926] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.217543] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 293.856717] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.859729] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 293.872684] Lustre: Skipped 2 previous similar messages [ 293.889609] Lustre: Skipped 1 previous similar message [ 298.977870] LustreError: 6531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.978541] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.998506] LustreError: 6531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 299.009447] Lustre: Skipped 1 previous similar message [ 299.522296] Lustre: server umount lustre-MDT0001 complete [ 302.928660] Lustre: server umount lustre-OST0000 complete [ 306.430667] Lustre: server umount lustre-OST0001 complete [ 313.871600] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 326.777079] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 359.117266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.356300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 359.382320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 359.473911] Lustre: lustre-MDT0000: new disk, initializing [ 359.554715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 359.572502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 364.128668] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 374.034751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.144942] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 374.229843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 374.234337] Lustre: Skipped 1 previous similar message [ 374.332960] Lustre: lustre-MDT0001: new disk, initializing [ 374.444166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.485686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.496251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 378.506488] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 387.809251] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 393.736577] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 400.609108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 400.942696] Lustre: lustre-OST0000: new disk, initializing [ 400.946538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 400.953122] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 401.062146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.356940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 402.366643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 402.478849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 407.203119] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 415.702309] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 419.926587] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 420.193082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 424.969447] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 425.201411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 428.001908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 428.011146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.021904] LustreError: 3660:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting [ 428.022348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 428.041757] Lustre: Skipped 1 previous similar message [ 438.242937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 438.250289] Lustre: Skipped 3 previous similar messages [ 441.311820] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 441.503586] Lustre: server umount lustre-OST0000 complete [ 446.433423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 446.444387] LustreError: Skipped 1 previous similar message [ 446.446920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.457776] Lustre: Skipped 1 previous similar message [ 451.445888] Lustre: server umount lustre-MDT0000 complete [ 453.609487] LustreError: 16457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 453.638557] LustreError: 16457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 455.022150] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438486 with bad export cookie 11874256787795258410 [ 455.027307] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 455.035131] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 455.392135] Lustre: server umount lustre-MDT0001 complete [ 461.918310] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:28:11 (1788438491) [ 481.033192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.400387] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.453608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.586143] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 486.882893] LustreError: 19931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.007691] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.708453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.549706] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 505.068281] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 510.240628] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 512.395179] Lustre: Failing over lustre-MDT0000 [ 512.409373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.415156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 512.427204] Lustre: Skipped 2 previous similar messages [ 512.441555] Lustre: Skipped 4 previous similar messages [ 514.642066] Lustre: server umount lustre-MDT0000 complete [ 517.607970] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.634581] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 533.813572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.921323] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.222309] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 538.229971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 538.268944] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 538.680542] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 548.907959] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 553.967731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 553.973217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.984652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 553.991079] Lustre: Skipped 2 previous similar messages [ 567.263138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 567.458054] Lustre: server umount lustre-MDT0000 complete [ 568.290176] LustreError: 19931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.305909] LustreError: 19931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 570.974784] LustreError: 19913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438601 with bad export cookie 11874256787795259824 [ 570.982918] LustreError: 19913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 570.984677] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 571.369210] Lustre: server umount lustre-MDT0001 complete [ 579.307488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.877662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.887682] Lustre: Skipped 2 previous similar messages [ 584.268366] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 592.901560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 598.883774] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 607.557347] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 614.234968] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 617.396586] Lustre: Failing over lustre-MDT0000 [ 617.834857] Lustre: server umount lustre-MDT0000 complete [ 618.472078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.478491] LustreError: 23265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.481540] Lustre: Skipped 2 previous similar messages [ 618.494925] LustreError: 23265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 634.848185] Lustre: 3660:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438649/real 1788438649] req@ffff9291b6c9bb80 x1875313263040256/t0(0) o400->MGC192.168.206.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788438665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 634.896530] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.159162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.114533] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa4c9d5a2d04eb55a [ 645.132191] Lustre: MGC192.168.206.157@tcp: Connection restored to 0@lo (at 0@lo) [ 645.136525] Lustre: Skipped 1 previous similar message [ 645.520398] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 650.734248] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 650.740283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 650.783498] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 651.202405] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 660.999529] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 666.080421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 666.093762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.115539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 666.123959] Lustre: Skipped 6 previous similar messages [ 671.641805] Lustre: server umount lustre-MDT0000 complete [ 675.589086] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438706 with bad export cookie 11874256787795260762 [ 675.591891] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 675.605526] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 675.872651] Lustre: server umount lustre-MDT0001 complete [ 685.241861] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:31:54 (1788438714) [ 702.002539] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 711.980377] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 749.202517] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 761.074703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.237377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 761.266344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 761.317230] Lustre: lustre-MDT0000: new disk, initializing [ 761.409308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.413098] Lustre: Skipped 2 previous similar messages [ 761.421949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 766.274931] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 777.011928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.154597] Lustre: 29640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 777.554237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 782.737168] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 792.299825] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 798.111227] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 804.559539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.855150] Lustre: 31608:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 806.667617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 806.703074] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 806.761075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 806.781800] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 810.477466] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 819.496203] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 827.753182] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 847.848959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 847.853554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.860682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 847.868484] Lustre: Skipped 2 previous similar messages [ 847.880564] Lustre: Skipped 2 previous similar messages [ 850.808354] Lustre: server umount lustre-OST0000 complete [ 860.402994] Lustre: server umount lustre-MDT0000 complete [ 863.204732] LustreError: 29653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 863.215859] LustreError: 29653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 863.804072] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438894 with bad export cookie 11874256787795261168 [ 863.809247] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 863.816371] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 864.390963] Lustre: server umount lustre-MDT0001 complete [ 874.551975] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 877.663217] Key type lgssc unregistered [ 878.070390] LNet: 33582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.082131] LNetError: 33582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.103059] LNet: Removed LNI 192.168.206.157@tcp [ 879.195168] Key type .llcrypt unregistered [ 879.199795] Key type ._llcrypt unregistered [ 903.652723] Key type ._llcrypt registered [ 903.655858] Key type .llcrypt registered [ 903.804982] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 905.350540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 905.388733] alg: No test for adler32 (adler32-zlib) [ 906.521476] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 906.771790] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 908.479175] Key type lgssc registered [ 909.488503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 918.082114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 918.110928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.847876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 925.537734] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 929.259201] LustreError: 34993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.287852] LustreError: 34993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 934.375338] LustreError: 34994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.587205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.899523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 939.599637] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 948.598887] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 954.811857] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 962.512325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 962.990379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.815138] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 977.266571] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 979.394602] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 997.025813] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:37:06 (1788439026) [ 999.361290] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1001.751423] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1009.997581] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:37:19 (1788439039) [ 1019.869813] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:37:29 (1788439049) [ 1021.305556] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1032.278786] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1032.283219] Lustre: Skipped 2 previous similar messages [ 1034.716730] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:37:43 (1788439063) [ 1037.006771] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1037.010132] Lustre: Skipped 2 previous similar messages [ 1045.724328] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1045.734034] Lustre: Skipped 4 previous similar messages [ 1060.419290] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:38:10 (1788439090) [ 1063.061734] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1065.739808] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1073.511995] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:38:23 (1788439103) [ 1075.439916] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1075.448133] Lustre: Skipped 6 previous similar messages [ 1089.414571] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:38:38 (1788439118) [ 1107.437842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1107.447344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.456685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1107.472924] Lustre: Skipped 1 previous similar message [ 1107.485033] Lustre: Skipped 1 previous similar message [ 1110.443823] Lustre: server umount lustre-OST0000 complete [ 1117.665312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.669728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1117.691823] Lustre: Skipped 1 previous similar message [ 1117.707903] Lustre: Skipped 1 previous similar message [ 1120.643450] Lustre: server umount lustre-MDT0000 complete [ 1122.787550] LustreError: 34988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.796784] LustreError: 34988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1124.122625] LustreError: 34975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439155 with bad export cookie 2152336437619930527 [ 1124.137811] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1124.627375] Lustre: server umount lustre-MDT0001 complete [ 1135.854822] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1138.254230] Key type lgssc unregistered [ 1138.526531] LNet: 42849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.540723] LNetError: 42849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.549377] LNet: Removed LNI 192.168.206.157@tcp [ 1139.153761] Key type .llcrypt unregistered [ 1139.155931] Key type ._llcrypt unregistered [ 1159.604249] Key type ._llcrypt registered [ 1159.606051] Key type .llcrypt registered [ 1159.698935] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 1177.287588] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1178.363767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1178.427892] alg: No test for adler32 (adler32-zlib) [ 1179.605788] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1179.889869] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1181.615160] Key type lgssc registered [ 1183.057179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1222.407060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1222.445308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.711189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1223.745372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1223.817967] Lustre: lustre-MDT0000: new disk, initializing [ 1223.904616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.923355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1228.787579] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1239.698885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.828407] Lustre: 45995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1239.873808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1239.877160] Lustre: Skipped 1 previous similar message [ 1239.959925] Lustre: lustre-MDT0001: new disk, initializing [ 1240.075932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1240.134609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1240.161546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1244.805568] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1254.842288] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1260.223610] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1267.900110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1268.332710] Lustre: lustre-OST0000: new disk, initializing [ 1268.337038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1268.349408] Lustre: 47963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1268.458963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.922735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1273.938501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1274.081702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1275.793728] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1285.784290] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1291.763294] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1292.077429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1297.546419] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1297.740370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1299.941325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.957188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1299.959087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1305.057964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.065355] Lustre: Skipped 2 previous similar messages [ 1310.181389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1310.200967] Lustre: Skipped 1 previous similar message [ 1314.271367] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1314.532514] Lustre: server umount lustre-OST0000 complete [ 1320.418846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1320.422680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1320.436609] Lustre: Skipped 2 previous similar messages [ 1320.454603] Lustre: Skipped 1 previous similar message [ 1325.176341] Lustre: server umount lustre-MDT0000 complete [ 1328.787283] LustreError: 47962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439359 with bad export cookie 730605205234244927 [ 1328.788634] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1328.800925] LustreError: 47962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1329.569787] Lustre: server umount lustre-MDT0001 complete [ 1340.631507] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:42:49 (1788439369) [ 1348.138962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.761371] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.778935] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1348.867343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.119567] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1354.212771] LustreError: 50429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.305862] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.004192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.352262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1368.297048] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1375.700277] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1381.984615] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1389.380083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.683713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.922741] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1403.730401] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1412.740330] LustreError: 52959:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1413.711599] Lustre: Modifying parameter general.timeout=1 in log params [ 1414.804450] Lustre: Modifying parameter general.timeout=2 in log params [ 1417.649447] Lustre: Modifying parameter general.timeout=5 in log params [ 1417.654152] Lustre: Skipped 2 previous similar messages [ 1422.273320] Lustre: Modifying parameter general.timeout=10 in log params [ 1422.283601] Lustre: Skipped 4 previous similar messages [ 1431.004452] Lustre: Modifying parameter general.timeout=19 in log params [ 1431.015645] Lustre: Skipped 8 previous similar messages [ 1447.045623] Lustre: Modifying parameter general.timeout=36 in log params [ 1447.054506] Lustre: Skipped 16 previous similar messages [ 1480.050270] Lustre: Modifying parameter general.timeout=68 in log params [ 1480.056768] Lustre: Skipped 31 previous similar messages [ 1549.777481] Lustre: Disabling parameter general.timeout= in log params [ 1549.783359] Lustre: Skipped 61 previous similar messages [ 1553.092917] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:46:22 (1788439582) [ 1556.388738] Lustre: 59907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1558.685575] Lustre: 60005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1562.444571] Lustre: 60216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1562.468264] Lustre: 60216:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1594.389616] Lustre: 61782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1594.399664] Lustre: 61782:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1594.424601] LustreError: 61782:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1598.161815] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:47:07 (1788439627) [ 1609.185892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1609.203068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.217182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.775945] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1612.799598] Lustre: Skipped 1 previous similar message [ 1615.063698] Lustre: server umount lustre-MDT0000 complete [ 1617.893210] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.904946] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1618.620231] LustreError: 50408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439649 with bad export cookie 730605205234245809 [ 1618.622926] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.629083] LustreError: 50408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1618.974301] Lustre: server umount lustre-MDT0001 complete [ 1632.282192] Lustre: server umount lustre-OST0000 complete [ 1664.488046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.806484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1665.111941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.909117] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1679.236546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.313898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1679.353155] Lustre: 64314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1683.824989] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1692.067695] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1697.861905] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1705.230129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1705.427563] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1705.748925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1705.758685] Lustre: Skipped 1 previous similar message [ 1712.798902] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1720.065785] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1721.838937] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1721.845405] Lustre: Skipped 23 previous similar messages [ 1721.883306] Lustre: 66883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1730.033423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1730.045523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.064594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1730.087541] Lustre: Skipped 3 previous similar messages [ 1734.412638] Lustre: server umount lustre-OST0000 complete [ 1740.772058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.792864] Lustre: Skipped 2 previous similar messages [ 1745.876695] Lustre: server umount lustre-MDT0000 complete [ 1745.900451] LustreError: 66187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1745.918320] LustreError: 66187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1749.973150] LustreError: 64308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439780 with bad export cookie 730605205234248875 [ 1749.975368] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1749.987428] LustreError: 64308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1750.242662] Lustre: server umount lustre-MDT0001 complete [ 1762.697649] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1767.042706] Key type lgssc unregistered [ 1767.314222] LNet: 68089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.322869] LNetError: 68089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.357587] LNet: Removed LNI 192.168.206.157@tcp [ 1768.406413] Key type .llcrypt unregistered [ 1768.409281] Key type ._llcrypt unregistered [ 1799.391514] Key type ._llcrypt registered [ 1799.393638] Key type .llcrypt registered [ 1799.550982] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1800.652912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1800.878082] alg: No test for adler32 (adler32-zlib) [ 1802.014825] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1802.339749] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1804.119153] Key type lgssc registered [ 1805.335651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1814.047112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1814.075223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.690285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.728778] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1825.251742] LustreError: 69499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1825.263493] LustreError: 69499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1829.667601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.344222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.838386] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1842.789546] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1848.066265] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1855.668208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1856.444486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.481598] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1871.094886] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1882.858562] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1895.395140] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.410326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1895.412505] Lustre: Skipped 1 previous similar message [ 1895.445820] Lustre: Skipped 2 previous similar messages [ 1900.095688] Lustre: server umount lustre-MDT0000 complete [ 1900.515558] LustreError: 69495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.535057] LustreError: 69495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1904.197754] LustreError: 69481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439935 with bad export cookie 12301420834055140514 [ 1904.199869] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1904.213433] LustreError: 69481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1904.747328] Lustre: server umount lustre-MDT0001 complete [ 1918.893817] Lustre: server umount lustre-OST0000 complete [ 1946.481275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.772119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1946.809793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1947.166478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.234746] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1962.945139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.056432] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1963.095527] Lustre: 74370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1963.642322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.168474] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1977.319782] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1983.802861] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1990.658045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1990.986807] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1991.403364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.539287] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2006.538765] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2008.571748] Lustre: 76946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2010.571242] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2018.788216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2018.802283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.818253] Lustre: Skipped 1 previous similar message [ 2018.827502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2019.299706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2021.939890] Lustre: server umount lustre-OST0000 complete [ 2029.540513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.556202] Lustre: Skipped 1 previous similar message [ 2029.567943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.574750] Lustre: Skipped 3 previous similar messages [ 2032.522028] Lustre: server umount lustre-MDT0000 complete [ 2034.656071] LustreError: 74382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.672724] LustreError: 74382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2036.469022] LustreError: 74362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440067 with bad export cookie 12301420834055141354 [ 2036.483701] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2036.485329] LustreError: 74362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2036.949165] Lustre: server umount lustre-MDT0001 complete [ 2049.462767] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2052.176205] Key type lgssc unregistered [ 2052.413627] LNet: 78157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.422520] LNetError: 78157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.447771] LNet: Removed LNI 192.168.206.157@tcp [ 2053.429866] Key type .llcrypt unregistered [ 2053.435207] Key type ._llcrypt unregistered [ 2083.393752] Key type ._llcrypt registered [ 2083.395410] Key type .llcrypt registered [ 2083.465186] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2084.470626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2084.530904] alg: No test for adler32 (adler32-zlib) [ 2085.502551] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2085.824688] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2087.567524] Key type lgssc registered [ 2089.188711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2096.946544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2096.977656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2098.452984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.105227] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2108.910361] LustreError: 79570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.929200] LustreError: 79570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2111.662535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.116799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2116.893481] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2126.056177] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2132.701577] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2140.352576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2140.881671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.848481] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2152.967902] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2163.138851] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2163.964590] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2165.044898] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2167.883644] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2167.889747] Lustre: Skipped 2 previous similar messages [ 2172.384926] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2172.393500] Lustre: Skipped 4 previous similar messages [ 2180.838774] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2180.850267] Lustre: Skipped 8 previous similar messages [ 2197.505564] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2197.508311] Lustre: Skipped 16 previous similar messages [ 2229.564388] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2229.572782] Lustre: Skipped 34 previous similar messages [ 2265.228683] Lustre: 87673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2268.133152] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:58:17 (1788440297) [ 2269.923774] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2271.785824] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:58:21 (1788440301) [ 2277.355290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2277.366196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.389541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.400430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.403789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.404171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.409690] Lustre: Skipped 1 previous similar message [ 2282.769794] Lustre: server umount lustre-OST0000 complete [ 2289.633374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.639442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.651622] Lustre: Skipped 1 previous similar message [ 2289.666738] Lustre: Skipped 1 previous similar message [ 2292.955983] Lustre: server umount lustre-MDT0000 complete [ 2294.769972] LustreError: 83411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.787201] LustreError: 83411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2296.857980] LustreError: 79550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440327 with bad export cookie 16955682366450273371 [ 2296.869835] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2296.874551] LustreError: 79550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2297.427646] Lustre: server umount lustre-MDT0001 complete [ 2308.093581] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2310.748710] Key type lgssc unregistered [ 2311.116731] LNet: 88922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.122682] LNetError: 88922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.156916] LNet: Removed LNI 192.168.206.157@tcp [ 2312.307147] Key type .llcrypt unregistered [ 2312.309181] Key type ._llcrypt unregistered [ 2316.926334] Key type ._llcrypt registered [ 2316.928488] Key type .llcrypt registered [ 2317.023276] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2321.807817] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules [ 2323.061647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2323.159276] alg: No test for adler32 (adler32-zlib) [ 2324.301778] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2324.573815] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2326.280653] Key type lgssc registered [ 2327.514095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2328.772962] LustreError: 90120:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2348.095281] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2350.930813] LustreError: 90120:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2357.110844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2357.143080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.716751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.212239] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2369.000928] LustreError: 91125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.034616] LustreError: 91125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2372.648708] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:00:01 (1788440401) [ 2374.114650] LustreError: 91124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.844852] LustreError: 91156:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2377.855265] LustreError: 91156:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2377.866733] LustreError: 91156:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2378.484322] Lustre: server umount lustre-MDT0000 complete [ 2393.633482] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2396.513697] Key type lgssc unregistered [ 2396.837765] LNet: 92649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.842603] LNetError: 92649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.862449] LNet: Removed LNI 192.168.206.157@tcp [ 2397.573863] Key type .llcrypt unregistered [ 2397.575749] Key type ._llcrypt unregistered [ 2415.593432] Key type ._llcrypt registered [ 2415.596279] Key type .llcrypt registered [ 2415.770973] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2417.113972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2417.259192] alg: No test for adler32 (adler32-zlib) [ 2418.526409] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2418.906969] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2420.671177] Key type lgssc registered [ 2421.883510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2430.440706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2430.480904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.254927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.255088] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2442.727909] LustreError: 93875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.763564] LustreError: 93875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2444.983269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.608692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2449.704823] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2457.351906] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2463.256157] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2469.561294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2470.297636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.667624] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2484.962568] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2533.025817] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:02:42 (1788440562) [ 2535.269905] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2537.603302] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:02:46 (1788440566) [ 2542.561604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2542.574388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.603902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2544.095960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2544.097518] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.792846] Lustre: server umount lustre-MDT0000 complete [ 2549.217658] LustreError: 94607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.226432] LustreError: 94607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2552.091984] LustreError: 93844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440583 with bad export cookie 963394231482752387 [ 2552.100526] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2552.110025] LustreError: 93844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2552.346556] Lustre: server umount lustre-MDT0001 complete [ 2567.068802] Lustre: server umount lustre-OST0000 complete [ 2581.058119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.525090] LustreError: 98147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.712747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.961882] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2587.109996] LustreError: 98148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.201385] LustreError: 98147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.152707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.115811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2603.434063] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2612.689501] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2618.773959] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2631.851072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.038529] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2632.043599] LustreError: 98134:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2632.063266] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2632.070632] LustreError: 100231:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2632.077829] LustreError: 100231:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2632.080946] LustreError: 100231:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2632.134748] Lustre: server umount lustre-OST0000 complete [ 2632.137257] LustreError: 100231:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2641.532569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2641.718373] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2641.724276] LustreError: 98134:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2641.754714] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2641.762186] LustreError: 100699:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2641.767254] LustreError: 100699:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2641.771537] LustreError: 100699:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2641.846820] Lustre: server umount lustre-OST0000 complete [ 2641.849576] LustreError: 100699:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2652.377947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.621752] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2652.631498] Lustre: Found index 0 for lustre-OST0000, updating log [ 2652.641594] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2652.829846] Lustre: lustre-OST0000: new disk, initializing [ 2652.836323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2652.842888] Lustre: 101235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2653.173747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.225838] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2664.388778] Lustre: Failing over lustre-OST0000 [ 2664.645475] Lustre: server umount lustre-OST0000 complete [ 2669.550561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2669.559872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.571214] Lustre: Skipped 1 previous similar message [ 2669.576261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2669.585044] Lustre: Skipped 2 previous similar messages [ 2672.614629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.615735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.659909] Lustre: Skipped 1 previous similar message [ 2674.769754] Lustre: server umount lustre-MDT0000 complete [ 2677.729364] LustreError: 98878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.739216] LustreError: 98878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2678.664227] LustreError: 98128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440709 with bad export cookie 963394231482755131 [ 2678.666064] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2678.685028] LustreError: 98128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2679.078964] Lustre: server umount lustre-MDT0001 complete [ 2689.208865] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:05:18 (1788440718) [ 2709.623905] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2721.180243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.684527] LustreError: 103583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.834650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.242351] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2738.557683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.500444] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2747.740967] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2755.675794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2756.650243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.657833] Lustre: Skipped 1 previous similar message [ 2763.900885] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2773.407541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.499467] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2773.605374] Lustre: lustre-OST0001: new disk, initializing [ 2773.610830] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2773.618581] Lustre: 105904:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2779.786551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2779.794725] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2779.907872] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2780.484353] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2787.883637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2795.187350] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2800.611125] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.622165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2800.625960] Lustre: Skipped 1 previous similar message [ 2800.636965] Lustre: Skipped 1 previous similar message [ 2805.933648] Lustre: server umount lustre-MDT0001 complete [ 2808.578391] LustreError: 103580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.589363] LustreError: 103580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2816.628372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.736087] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2816.738917] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2816.770712] Lustre: 103573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2818.618821] LustreError: 107481:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2818.634078] LustreError: 107481:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2818.652767] LustreError: 103576:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2818.721104] LustreError: 107487:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.157@tcp: cfg command failed: rc = -17 [ 2818.735551] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2818.735551] [ 2821.858988] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2822.645143] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2832.270322] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:07:41 (1788440861) [ 2849.415807] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2868.180858] Lustre: 109516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2893.249433] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2897.393845] Lustre: 110651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2991.073077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2991.076762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.082903] Lustre: Skipped 1 previous similar message [ 2991.086527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.089250] Lustre: Skipped 5 previous similar messages [ 2994.265071] Lustre: server umount lustre-MDT0000 complete [ 2996.704863] LustreError: 104818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.729812] LustreError: 104818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2998.304118] LustreError: 103565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441029 with bad export cookie 963394231482756125 [ 2998.313261] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2998.689948] Lustre: server umount lustre-MDT0001 complete [ 3013.854848] Lustre: server umount lustre-OST0000 complete [ 3017.183394] Lustre: 93174:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441032/real 1788441032] req@ffff92908c415500 x1875315732615296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3017.222464] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.245065] Lustre: Skipped 3 previous similar messages [ 3018.554323] Lustre: server umount lustre-OST0001 complete [ 3024.681634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3034.676745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3046.406211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3059.240746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3079.264220] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3089.613812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.638675] Lustre: lustre-MDT0000: reset Object Index mappings [ 3090.339262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.351845] Lustre: Skipped 2 previous similar messages [ 3095.102506] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3104.261131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.282122] Lustre: lustre-MDT0001: reset Object Index mappings [ 3109.637989] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3112.340573] Lustre: 115755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3118.231734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.079914] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3129.133078] LustreError: 116610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3129.145260] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3129.157735] LustreError: 116610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3135.159144] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3135.485309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3141.126013] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3141.130055] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3142.437860] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3151.335200] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3154.707046] Lustre: 117630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3209.184334] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:13:58 (1788441238) [ 3237.856085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3237.864404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.889747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3237.893215] Lustre: Skipped 3 previous similar messages [ 3241.547512] Lustre: server umount lustre-MDT0000 complete [ 3244.750473] LustreError: 114587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441275 with bad export cookie 963394231482803018 [ 3244.755412] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3244.768618] LustreError: 114587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3259.359868] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3259.601143] Lustre: server umount lustre-MDT0001 complete [ 3263.685078] LustreError: 118391:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x7c:0x0].0x0 (ffff92918705d300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3269.971513] Lustre: server umount lustre-OST0000 complete [ 3279.905102] Lustre: server umount lustre-OST0001 complete [ 3287.456958] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 3297.885495] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3334.033398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.239877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3334.270803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3334.336073] Lustre: lustre-MDT0000: new disk, initializing [ 3334.452696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.457732] Lustre: Skipped 3 previous similar messages [ 3334.471917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3339.230973] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3346.215841] Lustre: server umount lustre-MDT0000 complete [ 3356.003965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.409764] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3356.430799] Lustre: 121903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3356.461807] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3362.528775] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3367.729108] Lustre: server umount lustre-MDT0000 complete [ 3377.182624] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:16:46 (1788441406) [ 3378.790311] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3380.562680] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:16:50 (1788441410) [ 3398.846414] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3409.484934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.574886] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3423.489876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.649301] Lustre: 123952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3423.722681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3423.726953] Lustre: Skipped 1 previous similar message [ 3423.824075] Lustre: lustre-MDT0001: new disk, initializing [ 3423.988136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3424.012473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3427.998244] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3432.504943] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3439.221983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3439.449497] Lustre: lustre-OST0000: new disk, initializing [ 3439.453294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3439.463203] Lustre: 125529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3440.690688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3440.696780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3440.814425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3445.791371] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3457.186840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3457.353980] Lustre: lustre-OST0001: new disk, initializing [ 3457.359945] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3457.371259] Lustre: 126399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3458.837793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3458.852584] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3458.962736] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3463.230611] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3473.781238] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3478.233657] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3863.747817] Lustre: 132814:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3863.757826] Lustre: 132814:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3863.762776] Lustre: 132814:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3863.772060] Lustre: 132814:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3863.783184] Lustre: 132814:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3863.792892] Lustre: 132814:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4154.135735] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:29:43 (1788442183) [ 4171.370139] Lustre: server umount lustre-MDT0000 complete [ 4171.744373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4171.749208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4171.772931] Lustre: Skipped 5 previous similar messages [ 4171.789177] LustreError: 125524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.807452] LustreError: 125524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 4174.696148] LustreError: 125159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442205 with bad export cookie 963394231482872248 [ 4174.698484] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4174.714645] LustreError: 125159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4178.915593] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4178.918394] Lustre: Skipped 11 previous similar messages [ 4181.127769] Lustre: server umount lustre-MDT0001 complete [ 4190.677962] Lustre: server umount lustre-OST0000 complete [ 4208.095161] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4208.382587] Lustre: server umount lustre-OST0001 complete [ 4214.935339] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 4223.652864] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 4256.073517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4256.408932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4256.445941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4256.544562] Lustre: lustre-MDT0000: new disk, initializing [ 4256.635130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4256.643728] Lustre: Skipped 5 previous similar messages [ 4256.656829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4261.009304] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4270.830283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4270.941833] Lustre: 135869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4270.997592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4271.001416] Lustre: Skipped 1 previous similar message [ 4271.072602] Lustre: lustre-MDT0001: new disk, initializing [ 4271.174100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4271.190756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4274.994126] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4284.341881] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4290.390637] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4297.353849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4297.649611] Lustre: lustre-OST0000: new disk, initializing [ 4297.656582] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4297.668609] Lustre: 137836:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4297.772765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4297.783395] Lustre: Skipped 1 previous similar message [ 4299.081237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4299.102546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4299.213192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4303.648251] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4312.281346] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4323.211542] Lustre: lustre-MDD0000: changelog on [ 4325.776136] Lustre: lustre-MDD0001: changelog on [ 6251.634194] Lustre: lustre-MDD0001: changelog off [ 6253.765691] Lustre: 135875:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6255.059667] Lustre: lustre-MDD0000: changelog off [ 6265.313068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.322734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.350213] Lustre: Skipped 5 previous similar messages [ 6265.365784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6265.371674] Lustre: Skipped 3 previous similar messages [ 6268.221889] Lustre: server umount lustre-OST0000 complete [ 6272.998490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6273.017927] LustreError: Skipped 1 previous similar message [ 6273.022248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6273.029569] Lustre: Skipped 1 previous similar message [ 6273.032346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6275.553256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6275.557906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6277.790608] Lustre: server umount lustre-MDT0000 complete [ 6280.672529] LustreError: 138763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6280.690147] LustreError: 138763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6281.338846] LustreError: 135859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444312 with bad export cookie 963394231483690905 [ 6281.344298] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6281.348164] LustreError: 135859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6281.810251] Lustre: server umount lustre-MDT0001 complete [ 6293.899617] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6297.572336] Key type lgssc unregistered [ 6297.828872] LNet: 146996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6297.840958] LNetError: 146996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6297.860825] LNet: Removed LNI 192.168.206.157@tcp [ 6298.948359] Key type .llcrypt unregistered [ 6298.950869] Key type ._llcrypt unregistered [ 6321.737502] Key type ._llcrypt registered [ 6321.741333] Key type .llcrypt registered [ 6321.878334] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6339.061661] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6340.319781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6340.341634] alg: No test for adler32 (adler32-zlib) [ 6341.414198] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6341.700361] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6343.495159] Key type lgssc registered [ 6344.712690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6379.559614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6379.597801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6380.820871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6380.863608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6380.933458] Lustre: lustre-MDT0000: new disk, initializing [ 6380.998978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6381.012842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6387.539245] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6399.013721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6399.168262] Lustre: 150193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6399.224975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6399.229189] Lustre: Skipped 1 previous similar message [ 6399.305875] Lustre: lustre-MDT0001: new disk, initializing [ 6399.394078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6399.456043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6399.462697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6404.445969] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6413.755577] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6419.789189] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6426.352544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6426.614540] Lustre: lustre-OST0000: new disk, initializing [ 6426.618512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6426.625493] Lustre: 152163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6426.727993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6433.369764] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6433.868142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6433.885427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6434.024648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6443.255348] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6449.691655] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_module obdecho/obdecho [ 6451.298805] LustreError: 153167:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6451.316065] LustreError: 153167:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6453.036728] LustreError: 153263:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6464.999647] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6465.001015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6465.007026] Lustre: Skipped 2 previous similar messages [ 6465.011484] Lustre: Skipped 1 previous similar message [ 6469.783379] Lustre: server umount lustre-MDT0000 complete [ 6473.399266] LustreError: 150185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444504 with bad export cookie 17003551802775064331 [ 6473.401900] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6473.413509] LustreError: 150185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6473.678322] Lustre: server umount lustre-MDT0001 complete [ 6488.555452] Lustre: server umount lustre-OST0000 complete [ 6508.379232] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6518.815330] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6556.068198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.455466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6556.518323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6556.640719] Lustre: lustre-MDT0000: new disk, initializing [ 6556.738166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6556.756420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6562.902798] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6573.325708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6573.430208] Lustre: 156981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6573.498673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6573.503520] Lustre: Skipped 1 previous similar message [ 6573.554143] Lustre: lustre-MDT0001: new disk, initializing [ 6573.670940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6573.762917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6573.769694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6578.076108] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6588.116315] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6594.856606] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6602.664610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6602.915153] Lustre: lustre-OST0000: new disk, initializing [ 6602.917540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6602.920941] Lustre: 158946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6602.984993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6608.821400] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6612.543126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6612.553828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6612.650189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6617.632600] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6621.539949] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6621.731196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6625.810088] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6625.945927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6628.321119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6628.321996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6628.331848] LustreError: Skipped 1 previous similar message [ 6628.335886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6628.340942] Lustre: Skipped 1 previous similar message [ 6628.350176] Lustre: Skipped 1 previous similar message [ 6633.441242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6633.449803] Lustre: Skipped 1 previous similar message [ 6638.581281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6638.591164] Lustre: Skipped 1 previous similar message [ 6642.143118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6642.316589] Lustre: server umount lustre-OST0000 complete [ 6648.801073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6648.808708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6648.821472] Lustre: Skipped 1 previous similar message [ 6648.838069] Lustre: Skipped 1 previous similar message [ 6652.119160] Lustre: server umount lustre-MDT0000 complete [ 6653.921442] LustreError: 156992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.941307] LustreError: 156992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6655.602221] LustreError: 156972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444686 with bad export cookie 17003551802775065290 [ 6655.609121] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6655.621859] LustreError: 156972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6655.969170] Lustre: server umount lustre-MDT0001 complete [ 6659.733669] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:11:29 (1788444689) [ 6661.739353] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6663.832758] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:11:33 (1788444693) [ 6671.836272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6672.376236] LustreError: 161358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.454643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6677.478303] LustreError: 161359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6677.726219] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6682.595964] LustreError: 161358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.712818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6689.922427] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6698.643862] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6704.948135] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6712.156163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6712.467190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6712.476639] Lustre: Skipped 1 previous similar message [ 6719.239416] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6726.609449] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6737.739083] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.57@tcp (stopping) [ 6738.216121] Lustre: server umount lustre-MDT0000 complete [ 6741.477061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6741.485540] LustreError: 163851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6741.492734] Lustre: Skipped 2 previous similar messages [ 6741.514696] LustreError: 163851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6741.607133] LustreError: 161340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444772 with bad export cookie 17003551802775066172 [ 6741.609909] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6741.617283] LustreError: 161340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6746.604666] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6748.009797] Lustre: server umount lustre-MDT0001 complete [ 6754.775486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6767.956348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.274709] LustreError: 165200:0:(llog.c:1655:llog_backup()) MGC192.168.206.157@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6768.285430] Lustre: 165200:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6771.105202] LustreError: 165205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6771.131137] LustreError: 165205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6787.562139] LustreError: 165205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6787.591442] LustreError: 165205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6823.392146] LustreError: 165207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.415844] LustreError: 165207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 6829.023271] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6829.025738] Lustre: 165200:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.157@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6869.996590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6875.139284] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6875.809372] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6885.409296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6887.393713] LustreError: 148416:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004c4e5fdd x1875319845943168/t0(0) o250->MGC192.168.206.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6887.414720] LustreError: 165236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.446404] LustreError: 165236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6892.690236] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6893.056280] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6901.097531] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6902.764196] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6907.933272] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6909.683447] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6928.864749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6928.871256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6928.876947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6928.886550] Lustre: Skipped 2 previous similar messages [ 6928.894554] Lustre: Skipped 3 previous similar messages [ 6933.372292] Lustre: server umount lustre-MDT0000 complete [ 6936.522076] LustreError: 161340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444967 with bad export cookie 17003551802775072626 [ 6936.527911] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6936.540607] LustreError: 161340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6936.878019] Lustre: server umount lustre-MDT0001 complete [ 6950.678756] Lustre: server umount lustre-OST0000 complete [ 6971.308102] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6983.549728] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7026.248945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.463828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7026.494493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7026.584712] Lustre: lustre-MDT0000: new disk, initializing [ 7026.695957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7026.701226] Lustre: Skipped 1 previous similar message [ 7026.717275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7030.587957] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7040.056553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.181237] Lustre: 170305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7040.408362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7045.013401] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7053.582969] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7059.189377] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7066.208116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7066.514894] Lustre: lustre-OST0000: new disk, initializing [ 7066.517552] Lustre: Skipped 1 previous similar message [ 7066.532467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7066.539383] Lustre: Skipped 2 previous similar messages [ 7066.546821] Lustre: 172274:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7073.340552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7073.353346] Lustre: Skipped 1 previous similar message [ 7073.362557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7073.527282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7073.683858] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7084.404593] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7089.439659] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7089.617674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7093.831636] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7094.036969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7099.366119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7099.366314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7099.374280] LustreError: Skipped 1 previous similar message [ 7099.384656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7099.391814] Lustre: Skipped 1 previous similar message [ 7102.539640] Lustre: server umount lustre-OST0000 complete [ 7107.047488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7121.375250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7121.530826] Lustre: server umount lustre-MDT0000 complete [ 7124.872818] LustreError: 170295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445155 with bad export cookie 17003551802775089454 [ 7124.877875] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7124.885528] LustreError: 170295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7125.177584] Lustre: server umount lustre-MDT0001 complete [ 7134.162861] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:19:23 (1788445163) [ 7141.879819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7142.640660] LustreError: 174689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.668331] LustreError: 174689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7148.152383] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7157.653588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.818538] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7169.734440] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7174.646542] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7181.027780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7187.832725] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7198.209733] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7213.520305] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:20:41 (1788445241) [ 7219.685990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7219.711762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.752458] Lustre: Skipped 3 previous similar messages [ 7219.762264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7219.767472] Lustre: Skipped 7 previous similar messages [ 7225.408260] Lustre: server umount lustre-OST0000 complete [ 7236.262708] Lustre: server umount lustre-MDT0000 complete [ 7239.633039] LustreError: 174672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445270 with bad export cookie 17003551802775090336 [ 7239.641249] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7239.646092] LustreError: 174672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7239.993577] Lustre: server umount lustre-MDT0001 complete [ 7250.125923] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7252.804844] Key type lgssc unregistered [ 7253.071585] LNet: 178418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7253.084274] LNetError: 178418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7253.109278] LNet: Removed LNI 192.168.206.157@tcp [ 7253.956797] Key type .llcrypt unregistered [ 7253.959396] Key type ._llcrypt unregistered [ 7269.883583] Key type ._llcrypt registered [ 7269.885475] Key type .llcrypt registered [ 7270.012619] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7271.542028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7271.595377] alg: No test for adler32 (adler32-zlib) [ 7272.630078] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7272.911740] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 7274.559199] Key type lgssc registered [ 7275.700101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7281.995423] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7282.021848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7297.631954] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7302.751315] LustreError: 179637:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 7307.871743] LustreError: 179637:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7307.882243] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7307.889206] LustreError: MGC192.168.206.157@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7307.902201] LustreError: 179637:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7307.910233] LustreError: 179637:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7307.918687] LustreError: 179637:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7307.963949] Lustre: server umount lustre-OST0000 complete [ 7307.968658] LustreError: 179637:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7314.651566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.999114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7319.955858] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7325.157218] LustreError: 180135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7325.165822] LustreError: 180135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7327.695236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.139961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7332.229692] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7340.781722] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7346.997624] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7353.485599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7353.769612] LustreError: 181954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7353.831086] LustreError: 181950:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7353.850813] Lustre: 181950:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.157@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7354.096565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.642063] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7367.206717] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7370.721867] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.724280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7370.738933] Lustre: Skipped 2 previous similar messages [ 7370.752121] Lustre: Skipped 2 previous similar messages [ 7375.747298] Lustre: server umount lustre-MDT0000 complete [ 7375.845292] LustreError: 180129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7375.856703] LustreError: 180129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7379.476207] LustreError: 180115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445410 with bad export cookie 17286465939950932601 [ 7379.486281] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7379.799908] Lustre: server umount lustre-MDT0001 complete [ 7395.362903] Lustre: server umount lustre-OST0000 complete [ 7408.874383] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7421.252636] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7461.147702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.332441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7461.357553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7461.482298] Lustre: lustre-MDT0000: new disk, initializing [ 7461.635896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.668919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7466.790844] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7478.210445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7478.336679] Lustre: 185477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7478.395929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7478.399751] Lustre: Skipped 1 previous similar message [ 7478.487534] Lustre: lustre-MDT0001: new disk, initializing [ 7478.640873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7478.713557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7478.726304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7483.849857] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7494.040482] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7500.798910] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7508.968359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7509.225757] Lustre: lustre-OST0000: new disk, initializing [ 7509.229356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7509.240914] Lustre: 187445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7509.347399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7517.296143] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7519.359728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7519.380245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7519.536942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7528.500989] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7533.997948] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7534.329029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7538.482499] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7538.773910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7545.316164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.319718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.326633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.356741] Lustre: Skipped 1 previous similar message [ 7547.603752] Lustre: server umount lustre-OST0000 complete [ 7555.555639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7555.568162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7555.576226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7555.587240] Lustre: Skipped 1 previous similar message [ 7555.600241] Lustre: Skipped 2 previous similar messages [ 7558.470569] Lustre: server umount lustre-MDT0000 complete [ 7560.674197] LustreError: 185488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.696373] LustreError: 185488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7562.562861] LustreError: 185471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445593 with bad export cookie 17286465939950933322 [ 7562.577745] LustreError: 185471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7562.585883] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7563.068993] Lustre: server umount lustre-MDT0001 complete [ 7575.286799] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:26:44 (1788445604) [ 7596.074366] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7599.482428] Key type lgssc unregistered [ 7599.754491] LNet: 190391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7599.764441] LNetError: 190391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7599.789743] LNet: Removed LNI 192.168.206.157@tcp [ 7600.760379] Key type .llcrypt unregistered [ 7600.762588] Key type ._llcrypt unregistered [ 7620.869221] Key type ._llcrypt registered [ 7620.876628] Key type .llcrypt registered [ 7621.054562] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7622.354278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7622.495976] alg: No test for adler32 (adler32-zlib) [ 7623.561616] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7623.827171] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 7625.591244] Key type lgssc registered [ 7626.886098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7635.889549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7635.913480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.372854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7643.105994] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7647.728376] LustreError: 191604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.768115] LustreError: 191604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7652.844320] LustreError: 191605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7652.883275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7653.164972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7658.319808] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7666.847973] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7673.570251] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7682.562915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7682.734392] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7682.738264] LustreError: 193472:0:(llog.c:1696:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7682.743242] Lustre: 193472:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.157@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7682.870958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.568261] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7697.743765] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7703.523747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.530180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7703.536548] Lustre: Skipped 1 previous similar message [ 7707.074461] Lustre: server umount lustre-MDT0000 complete [ 7708.656490] LustreError: 191604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.676671] LustreError: 191604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7711.518274] LustreError: 191584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445742 with bad export cookie 10782190775740501311 [ 7711.524490] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7711.529017] LustreError: 191584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7712.020933] Lustre: server umount lustre-MDT0001 complete [ 7726.925629] Lustre: server umount lustre-OST0000 complete [ 7735.885507] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7747.722194] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7787.042259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7787.324946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7787.368816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7787.466797] Lustre: lustre-MDT0000: new disk, initializing [ 7787.574183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7787.595482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7791.905558] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7801.843081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7801.943041] Lustre: 196999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7802.032042] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7802.034467] Lustre: Skipped 1 previous similar message [ 7802.041923] LustreError: 197781:0:(llog.c:1696:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7802.050167] LustreError: 197781:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7802.057020] Lustre: 197781:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7802.075534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7802.079629] Lustre: Skipped 1 previous similar message [ 7802.147873] Lustre: lustre-MDT0001: new disk, initializing [ 7802.240512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7802.269665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7802.276134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7806.667933] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7815.144729] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7819.955923] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7826.164830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7826.431355] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7826.433568] Lustre: Skipped 1 previous similar message [ 7826.435997] LustreError: 198944:0:(llog.c:1696:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7826.444387] LustreError: 198944:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7826.451609] Lustre: 198944:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7826.457653] Lustre: 198944:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7826.475489] Lustre: lustre-OST0000: new disk, initializing [ 7826.483819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7826.488647] Lustre: 198967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7826.565043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7832.148276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7832.164044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7832.323770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7832.556514] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7841.886453] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7846.216778] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7846.414512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7850.437848] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7850.686961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7853.024195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7853.026977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.036861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7853.048370] Lustre: Skipped 2 previous similar messages [ 7853.056665] Lustre: Skipped 2 previous similar messages [ 7858.158796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7858.171018] Lustre: Skipped 2 previous similar messages [ 7858.464119] Lustre: server umount lustre-OST0000 complete [ 7863.269674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.275975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7863.284985] Lustre: Skipped 1 previous similar message [ 7868.385791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7868.397557] Lustre: Skipped 2 previous similar messages [ 7869.156440] Lustre: server umount lustre-MDT0000 complete [ 7873.509808] LustreError: 197010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7873.523788] LustreError: 196992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445904 with bad export cookie 10782190775740502039 [ 7873.538222] LustreError: 197010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7873.543693] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7873.553926] LustreError: 196992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7873.798626] Lustre: server umount lustre-MDT0001 complete [ 7882.367304] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:31:52 (1788445912) [ 7899.255709] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7908.716491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.109445] LustreError: 202094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.152439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7913.523929] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7919.585827] LustreError: 202094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7919.611732] LustreError: 202094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7923.211743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7929.599325] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7933.354994] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7940.416834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7948.264878] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7956.270723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7956.447755] Lustre: lustre-OST0001: new disk, initializing [ 7956.450597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7956.454941] Lustre: 204406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7956.523430] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7956.526645] Lustre: Skipped 2 previous similar messages [ 7963.545912] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7964.874696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7964.899540] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7965.046490] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7974.197919] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7983.579559] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7985.554285] Lustre: DEBUG MARKER: ADD OST3 [ 7991.666041] loop: module loaded [ 7991.729394] loop0: detected capacity change from 0 to 204800000 [ 7991.936248] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7991.955326] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7991.987394] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7998.958340] loop0: detected capacity change from 0 to 204800000 [ 7999.347277] Lustre: DEBUG MARKER: STOP OST3 [ 8005.950631] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8006.188602] Lustre: lustre-OST0003: new disk, initializing [ 8006.200378] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8006.211360] Lustre: 206451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8007.538288] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8007.668088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8007.689108] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8007.729419] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8009.223967] Lustre: Failing over lustre-OST0003 [ 8009.571315] LustreError: 206571:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8009.635731] Lustre: server umount lustre-OST0003 complete [ 8012.580327] LustreError: 203584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8012.786267] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8012.787248] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8013.217868] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8020.125476] Lustre: DEBUG MARKER: START OST3 again [ 8026.304271] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8026.603801] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8026.610171] Lustre: Skipped 1 previous similar message [ 8026.640025] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8027.903223] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8028.417972] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8028.418296] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8032.609268] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8053.844113] Lustre: Failing over lustre-OST0003 [ 8053.971595] Lustre: server umount lustre-OST0003 complete [ 8054.249582] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8054.251812] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8054.253409] LustreError: Skipped 1 previous similar message [ 8054.258043] LustreError: 203581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8054.266242] Lustre: Skipped 2 previous similar messages [ 8054.290851] LustreError: 203581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 8072.168569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8072.173462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8072.180817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8078.366256] Lustre: server umount lustre-MDT0000 complete [ 8081.052614] LustreError: 202076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446112 with bad export cookie 10782190775740502921 [ 8081.064059] LustreError: 202076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8081.072417] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8087.474995] Lustre: server umount lustre-MDT0001 complete [ 8090.891992] Lustre: server umount lustre-OST0000 complete [ 8094.184829] Lustre: server umount lustre-OST0001 complete [ 8099.963528] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 8107.114693] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8143.319723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8143.527239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8143.554209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8143.674742] Lustre: lustre-MDT0000: new disk, initializing [ 8143.758661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8148.271294] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8159.068514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8159.203470] Lustre: 211871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8159.386157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8159.398045] Lustre: Skipped 1 previous similar message [ 8159.449046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8164.083210] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8174.241322] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8181.518564] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8190.146570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8190.453212] Lustre: lustre-OST0000: new disk, initializing [ 8190.461284] Lustre: Skipped 1 previous similar message [ 8190.467210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8190.472114] Lustre: Skipped 2 previous similar messages [ 8190.479622] Lustre: 213840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8192.092761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8192.113811] Lustre: Skipped 1 previous similar message [ 8192.117166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8192.242729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8197.481967] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8208.265701] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8213.202892] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8213.384135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8218.303981] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8218.546480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8223.207492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8223.210537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8223.213239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8223.213248] Lustre: Skipped 11 previous similar messages [ 8223.239619] Lustre: Skipped 6 previous similar messages [ 8226.613630] Lustre: server umount lustre-OST0000 complete [ 8231.392304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8231.406316] LustreError: Skipped 1 previous similar message [ 8236.577777] Lustre: server umount lustre-MDT0000 complete [ 8238.562060] LustreError: 211882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8238.574023] LustreError: 211882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 8239.744834] LustreError: 211864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446270 with bad export cookie 10782190775740505210 [ 8239.749222] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8239.754311] LustreError: 211864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8240.073190] Lustre: server umount lustre-MDT0001 complete [ 8243.294792] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:37:52 (1788446272) [ 8260.008628] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 8269.590583] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8305.160547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8305.428792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8305.461739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8305.578089] Lustre: lustre-MDT0000: new disk, initializing [ 8305.660635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8309.847817] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8319.352079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8319.499045] Lustre: 218453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8319.828837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8324.283459] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8333.078876] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8338.825270] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8345.054716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8345.393209] Lustre: 220419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8346.838640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8351.906507] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8361.147539] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8365.280617] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8365.469300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8370.180739] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8370.429900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8372.706718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8372.708750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8372.727802] Lustre: Skipped 3 previous similar messages [ 8372.740150] Lustre: Skipped 5 previous similar messages [ 8378.751373] Lustre: server umount lustre-OST0000 complete [ 8386.537888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8389.687246] Lustre: server umount lustre-MDT0000 complete [ 8393.190177] LustreError: 218465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8393.208446] LustreError: 218465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8393.607840] LustreError: 219247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446424 with bad export cookie 10782190775740506099 [ 8393.621512] LustreError: 219247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8393.623956] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8393.985075] Lustre: server umount lustre-MDT0001 complete [ 8402.802117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.237512] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8415.507049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8415.829119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8415.831152] Lustre: Skipped 5 previous similar messages [ 8419.947873] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8427.747812] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8432.911488] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8438.865802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8444.879556] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8451.660777] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8463.846483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8463.857270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.889029] Lustre: Skipped 2 previous similar messages [ 8463.891472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8463.900559] Lustre: Skipped 5 previous similar messages [ 8467.018250] Lustre: server umount lustre-OST0000 complete [ 8476.715307] Lustre: server umount lustre-MDT0000 complete [ 8480.843332] LustreError: 222621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446511 with bad export cookie 10782190775740506981 [ 8480.855170] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8481.283160] Lustre: server umount lustre-MDT0001 complete [ 8589.025455] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 8591.343234] Key type lgssc unregistered [ 8591.661655] LNet: 226302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8591.673958] LNetError: 226302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8591.690734] LNet: Removed LNI 192.168.206.157@tcp [ 8592.470174] Key type .llcrypt unregistered [ 8592.474242] Key type ._llcrypt unregistered [ 8614.353713] Key type ._llcrypt registered [ 8614.356518] Key type .llcrypt registered [ 8614.487894] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 8628.287476] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8629.593697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8629.662129] alg: No test for adler32 (adler32-zlib) [ 8630.904985] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8631.282593] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 8633.071234] Key type lgssc registered [ 8634.447580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8669.485339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8669.533233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.796426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8670.819201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8670.876978] Lustre: lustre-MDT0000: new disk, initializing [ 8670.951936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8670.966504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8675.877946] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8687.355798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.542534] Lustre: 229643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8687.603293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8687.608715] Lustre: Skipped 1 previous similar message [ 8687.699892] Lustre: lustre-MDT0001: new disk, initializing [ 8687.819903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8687.876283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8687.888312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8692.931902] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8702.856859] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8708.897449] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8715.609323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8715.872690] Lustre: lustre-OST0000: new disk, initializing [ 8715.875784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8715.896145] Lustre: 231612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8715.969879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8722.317929] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8726.079197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8726.085573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8726.191190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8731.467323] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8735.819679] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8736.019615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8740.208987] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8740.398903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8746.981552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8746.981617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8746.991354] LustreError: Skipped 1 previous similar message [ 8746.995431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8747.005030] Lustre: Skipped 1 previous similar message [ 8747.022865] Lustre: Skipped 1 previous similar message [ 8752.102669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8752.110319] Lustre: Skipped 1 previous similar message [ 8756.191461] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8756.328424] Lustre: server umount lustre-OST0000 complete [ 8759.783767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8759.788662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8759.805165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8762.336835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8762.341489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.981582] Lustre: server umount lustre-MDT0000 complete [ 8771.498778] LustreError: 229635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446802 with bad export cookie 13889787616439542203 [ 8771.505129] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8771.513342] LustreError: 229635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8772.016798] Lustre: server umount lustre-MDT0001 complete [ 8788.781907] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8798.508662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8799.143656] LustreError: 234545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8799.176592] LustreError: 234545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8799.231371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8804.328544] LustreError: 234546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8805.334878] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8809.461296] LustreError: 234545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8814.512884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.578127] LustreError: 234546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8814.898170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8818.746887] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8821.399618] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8827.764782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8828.172759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8834.652860] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8843.263335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8843.427405] Lustre: lustre-OST0001: new disk, initializing [ 8843.430625] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8843.437948] Lustre: 236855:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8849.461414] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8851.033733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8851.052445] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8851.144785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8860.551864] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8869.662807] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8882.036410] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:48:31 (1788446911) [ 8886.755950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8886.780429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.801168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8886.812773] Lustre: Skipped 1 previous similar message [ 8890.408338] Lustre: server umount lustre-MDT0000 complete [ 8892.391802] LustreError: 234540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.404591] LustreError: 234540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8893.657712] LustreError: 235680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446924 with bad export cookie 13889787616439543085 [ 8893.662862] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8893.665423] LustreError: 235680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8893.977239] Lustre: server umount lustre-MDT0001 complete [ 8907.874555] Lustre: server umount lustre-OST0000 complete [ 8922.715781] Lustre: server umount lustre-OST0001 complete [ 8928.884729] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 8939.321911] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 8972.984195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8973.244891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8973.270853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8973.385489] Lustre: lustre-MDT0000: new disk, initializing [ 8973.536552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8973.543060] Lustre: Skipped 1 previous similar message [ 8973.557634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8978.396676] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 8988.868409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.016167] Lustre: 240799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8989.115451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8989.124197] Lustre: Skipped 1 previous similar message [ 8989.215392] Lustre: lustre-MDT0001: new disk, initializing [ 8989.339677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8989.349895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8993.751224] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9003.155564] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9008.714819] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9016.030953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9016.395207] Lustre: lustre-OST0000: new disk, initializing [ 9016.399993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9016.413461] Lustre: 242770:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9016.543378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9016.552146] Lustre: Skipped 1 previous similar message [ 9018.281501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9018.291970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9018.408120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9023.714764] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9033.257629] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9038.088188] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9038.396248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9043.176977] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9043.377822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9049.056899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9049.067301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9049.072416] LustreError: Skipped 1 previous similar message [ 9049.074678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9049.074685] Lustre: Skipped 4 previous similar messages [ 9049.098801] Lustre: Skipped 4 previous similar messages [ 9051.493355] Lustre: server umount lustre-OST0000 complete [ 9055.712590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9059.297780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.319604] Lustre: Skipped 1 previous similar message [ 9061.196856] Lustre: server umount lustre-MDT0000 complete [ 9064.416922] LustreError: 240813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9064.430597] LustreError: 240813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9064.560783] LustreError: 240792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447095 with bad export cookie 13889787616439545220 [ 9064.566789] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9064.575844] LustreError: 240792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9064.852514] Lustre: server umount lustre-MDT0001 complete [ 9072.526762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.508750] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9082.336735] LustreError: 244988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9082.350130] LustreError: 244988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9084.319033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9084.577131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9084.584536] Lustre: Skipped 1 previous similar message [ 9089.244965] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9096.611704] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9102.426369] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9108.644406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9114.865140] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9122.386459] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9268.709874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9268.721755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9268.745657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9268.756820] Lustre: Skipped 4 previous similar messages [ 9273.902974] Lustre: server umount lustre-OST0000 complete [ 9283.862653] Lustre: server umount lustre-MDT0000 complete [ 9286.116230] LustreError: 244988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9286.132677] LustreError: 244988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9287.627994] LustreError: 244969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447318 with bad export cookie 13889787616439546102 [ 9287.644852] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9287.649588] LustreError: 244969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9288.155507] Lustre: server umount lustre-MDT0001 complete [ 9392.558574] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 9395.532855] Key type lgssc unregistered [ 9395.822293] LNet: 248598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9395.828457] LNetError: 248598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9395.847759] LNet: Removed LNI 192.168.206.157@tcp [ 9396.766151] Key type .llcrypt unregistered [ 9396.768912] Key type ._llcrypt unregistered [ 9418.925325] Key type ._llcrypt registered [ 9418.930694] Key type .llcrypt registered [ 9419.054289] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 9431.077651] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9431.945906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9431.961598] alg: No test for adler32 (adler32-zlib) [ 9433.024176] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9433.240400] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 9434.889472] Key type lgssc registered [ 9435.731182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9478.688466] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9487.126547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9487.158586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9488.387757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9488.418471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9488.515225] Lustre: lustre-MDT0000: new disk, initializing [ 9488.622150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9488.630770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9493.082673] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9503.790626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9503.881073] Lustre: 252642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9503.961797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9503.973140] Lustre: Skipped 1 previous similar message [ 9504.104303] Lustre: lustre-MDT0001: new disk, initializing [ 9504.227717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9504.256644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9504.272428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9508.718730] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9513.430655] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9520.416922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9520.716077] Lustre: lustre-OST0000: new disk, initializing [ 9520.719546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9520.730583] Lustre: 254275:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9520.876351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9521.449876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9521.471750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9521.638029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9527.046553] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9537.949367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9538.083874] Lustre: lustre-OST0001: new disk, initializing [ 9538.089404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9538.099952] Lustre: 255145:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9538.220373] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9545.091429] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9547.357173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9547.378220] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9547.476929] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9557.321668] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9565.367918] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9578.466558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9578.472578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9578.495662] Lustre: Skipped 3 previous similar messages [ 9578.506891] Lustre: Skipped 3 previous similar messages [ 9582.857851] Lustre: server umount lustre-MDT0000 complete [ 9586.876292] LustreError: 252633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447617 with bad export cookie 11596554374868092203 [ 9586.887829] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9586.892737] LustreError: 252633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9587.311561] Lustre: server umount lustre-MDT0001 complete [ 9601.475854] Lustre: server umount lustre-OST0000 complete [ 9604.065113] Lustre: 250150:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447619/real 1788447619] req@ffff929083827b80 x1875323088089472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447635 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9604.087432] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.746899] Lustre: server umount lustre-OST0001 complete [ 9618.244302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9627.178794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9634.830790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9643.568090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9664.145103] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9672.798952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9672.822847] Lustre: lustre-MDT0000: reset Object Index mappings [ 9673.142744] LustreError: 260166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9673.157875] LustreError: 260166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9673.215529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9677.165151] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9678.306627] LustreError: 260167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9683.431935] LustreError: 260166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9684.461694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9684.491678] Lustre: lustre-MDT0001: reset Object Index mappings [ 9684.951503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9689.284114] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9691.925790] Lustre: 261308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9697.499240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.882265] LustreError: 261662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9698.906177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9703.922167] LustreError: 261662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.986130] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9712.161442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9712.399307] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9712.410864] Lustre: Skipped 1 previous similar message [ 9717.753101] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9718.183811] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9725.365937] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9729.064848] Lustre: 263177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9736.182017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9736.186197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9736.253795] LustreError: 260166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9736.283798] LustreError: 260166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9736.311848] Lustre: server umount lustre-MDT0000 complete [ 9738.223509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9738.249976] Lustre: Skipped 1 previous similar message [ 9740.575668] LustreError: 260910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447771 with bad export cookie 11596554374868099000 [ 9740.584409] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9740.592407] LustreError: 260910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9741.057770] Lustre: server umount lustre-MDT0001 complete [ 9755.647618] Lustre: server umount lustre-OST0000 complete [ 9769.577621] Lustre: server umount lustre-OST0001 complete [ 9792.620602] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:03:41 (1788447821) [ 9810.149780] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 9821.275743] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9857.691680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9858.018330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9858.050161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9858.159200] Lustre: lustre-MDT0000: new disk, initializing [ 9858.287701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9858.365031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9863.777872] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9875.158133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9875.260977] Lustre: 267432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9875.300951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9875.303962] Lustre: Skipped 1 previous similar message [ 9875.357215] Lustre: lustre-MDT0001: new disk, initializing [ 9875.469390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9875.479908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9880.145333] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9888.994164] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9894.191660] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9901.385695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9901.623698] Lustre: lustre-OST0000: new disk, initializing [ 9901.626615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9901.631743] Lustre: 269403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9902.943479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9902.952634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9903.045136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9907.977718] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 9918.271543] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9923.123918] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9923.332448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9928.399219] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9928.581247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9933.795320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9933.802910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9933.812967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9938.917182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9938.922913] Lustre: Skipped 2 previous similar messages [ 9944.034780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9945.055126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9945.344103] Lustre: server umount lustre-OST0000 complete [ 9952.224572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9952.240545] LustreError: Skipped 1 previous similar message [ 9952.247804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9952.274518] Lustre: Skipped 1 previous similar message [ 9952.283171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9952.291904] Lustre: Skipped 1 previous similar message [ 9955.663795] Lustre: server umount lustre-MDT0000 complete [ 9959.095739] LustreError: 267424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447990 with bad export cookie 11596554374868101093 [ 9959.105889] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9959.389365] Lustre: server umount lustre-MDT0001 complete [ 9978.436652] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 9989.982628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9990.398911] LustreError: 272340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9990.430788] LustreError: 272340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9990.472728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9990.479910] Lustre: Skipped 2 previous similar messages [ 9995.943311] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10004.484917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.267584] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10012.745088] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10020.886275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10028.057345] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10036.481511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10036.639835] Lustre: lustre-OST0001: new disk, initializing [10036.642926] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10036.648952] Lustre: 274646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10038.571387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10038.579452] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10038.735553] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10042.871432] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10052.106439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10056.173421] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10064.041741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10064.075625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10079.721458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.730824] Lustre: Skipped 3 previous similar messages [10079.733985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10079.748460] Lustre: Skipped 5 previous similar messages [10083.561759] Lustre: server umount lustre-MDT0000 complete [10084.835484] LustreError: 273068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10084.843666] LustreError: 273068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10087.184274] LustreError: 272321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448118 with bad export cookie 11596554374868101975 [10087.186130] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10087.196797] LustreError: 272321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10087.564292] Lustre: server umount lustre-MDT0001 complete [10101.289946] Lustre: server umount lustre-OST0000 complete [10115.721631] Lustre: server umount lustre-OST0001 complete [10133.819874] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10146.097462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10146.461821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10146.468814] Lustre: Skipped 3 previous similar messages [10151.340478] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10151.907671] LustreError: 277994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10151.936388] LustreError: 277994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10159.480609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.708264] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10167.790366] Lustre: 279136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10174.323579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10180.972698] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10188.288670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10192.648989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10192.704557] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10194.116898] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10201.133280] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10204.603085] Lustre: 281011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10209.404803] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10209.526185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10221.657858] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:10:51 (1788448251) [10242.018704] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10242.020451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10242.026374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10242.045368] Lustre: Skipped 2 previous similar messages [10244.017112] Lustre: server umount lustre-MDT0000 complete [10248.087828] LustreError: 278737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448279 with bad export cookie 11596554374868108835 [10248.098241] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10248.100938] LustreError: 278737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10254.560803] Lustre: server umount lustre-MDT0001 complete [10273.247231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10273.385256] Lustre: server umount lustre-OST0000 complete [10277.452933] Lustre: server umount lustre-OST0001 complete [10285.306967] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [10296.137379] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10348.959897] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10359.110585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.405513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10359.425475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10359.485551] Lustre: lustre-MDT0000: new disk, initializing [10359.555961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10365.595312] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10378.468483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10378.653495] Lustre: 285542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10379.138433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10384.945365] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10396.211663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10396.437347] Lustre: 287173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10398.162679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10403.414460] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10416.298200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10416.521435] Lustre: 288046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10416.648036] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10416.653512] Lustre: Skipped 6 previous similar messages [10418.146756] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10418.160823] Lustre: Skipped 1 previous similar message [10418.332388] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10424.224381] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10433.991535] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10437.337990] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10437.345290] Lustre: Skipped 1 previous similar message [10452.258180] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:14:41 (1788448481) [10456.035107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10456.049401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10456.061623] Lustre: Skipped 2 previous similar messages [10456.075095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10456.081124] Lustre: Skipped 7 previous similar messages [10456.517627] Lustre: server umount lustre-MDT0000 complete [10459.113733] LustreError: 285553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.128142] LustreError: 285553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10460.338287] LustreError: 285534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448491 with bad export cookie 11596554374868116325 [10460.346323] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10460.357493] LustreError: 285534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10460.836393] Lustre: server umount lustre-MDT0001 complete [10474.967620] Lustre: server umount lustre-OST0000 complete [10488.573795] Lustre: server umount lustre-OST0001 complete [10496.409477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10500.781031] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10508.384044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.121392] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10520.001330] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10525.531977] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10532.902315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10540.132259] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10548.726444] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10566.723370] loop1: detected capacity change from 0 to 204800000 [10566.988558] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10567.007557] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10567.026079] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10572.374811] loop1: detected capacity change from 0 to 204800000 [10578.533329] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10578.669175] LustreError: 290281:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs6884: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10578.678412] LustreError: 290281:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs6884 [10578.685559] LustreError: 290281:0:(mgs_handler.c:517:mgs_target_reg()) fs6884-MDT0001: Error checking target index: rc = -13 [10578.705577] LustreError: fs6884-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10578.716888] LustreError: 294012:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10578.735770] LustreError: 294012:0:(tgt_mount.c:2132:server_put_super()) no obd fs6884-MDT0001 [10578.739341] LustreError: 294012:0:(tgt_mount.c:116:server_deregister_mount()) fs6884-MDT0001 not registered [10578.760859] Lustre: server umount fs6884-MDT0001 complete [10578.763242] LustreError: 294012:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10589.248619] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10589.514857] Lustre: Setting parameter fs6884-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs6884-MDT0001 [10589.596599] Lustre: srv-fs6884-MDT0001: No data found on store. Initialize space. [10589.600931] Lustre: Skipped 4 previous similar messages [10589.701834] Lustre: fs6884-MDT0001: new disk, initializing [10589.705557] Lustre: Skipped 3 previous similar messages [10589.917329] Lustre: srv-fs6884-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10589.936305] Lustre: 294479:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6884-MDT0001-mdtlov: can't create new llog: rc = -115 [10589.955811] LustreError: 294479:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6884-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10595.321501] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10603.523902] Lustre: server umount fs6884-MDT0001 complete [10618.344418] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10619.550194] loop1: detected capacity change from 0 to 204800000 [10626.805035] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.258328] Lustre: 296048:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6884-MDT0001-mdtlov: can't create new llog: rc = -115 [10627.266788] LustreError: 296048:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6884-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10632.485743] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10637.786646] Lustre: server umount fs6884-MDT0001 complete [10649.063650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10649.065631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10649.074174] Lustre: Skipped 5 previous similar messages [10649.077714] Lustre: Skipped 1 previous similar message [10655.193520] Lustre: server umount lustre-MDT0000 complete [10665.013643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10665.182452] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10670.589751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10671.308697] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10680.217450] loop1: detected capacity change from 0 to 204800000 [10685.756063] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.178799] Lustre: 298498:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6884-MDT0001-mdtlov: can't create new llog: rc = -115 [10686.202129] LustreError: 298498:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6884-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10690.984351] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10695.747295] Lustre: server umount fs6884-MDT0001 complete [10716.657575] LustreError: 292122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10716.681173] LustreError: 292122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 113 previous similar messages [10718.688388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10718.696087] LustreError: Skipped 1 previous similar message [10730.975135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10731.212807] Lustre: server umount lustre-OST0000 complete [10741.157902] Lustre: server umount lustre-MDT0000 complete [10744.978139] LustreError: 290273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448775 with bad export cookie 11596554374868119006 [10744.985588] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10744.990574] LustreError: 290273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10745.617405] Lustre: server umount lustre-MDT0001 complete [10758.681522] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [10761.968524] Key type lgssc unregistered [10762.357835] LNet: 300842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10762.370388] LNetError: 300842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10762.388309] LNet: Removed LNI 192.168.206.157@tcp [10763.155167] Key type .llcrypt unregistered [10763.159945] Key type ._llcrypt unregistered [10786.112554] Key type ._llcrypt registered [10786.115179] Key type .llcrypt registered [10786.206304] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [10787.883387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10787.974879] alg: No test for adler32 (adler32-zlib) [10789.264506] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10789.619038] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [10791.407901] Key type lgssc registered [10793.142292] Lustre: Echo OBD driver; http://www.lustre.org/ [10800.992312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10801.045878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10802.344134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10807.495377] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10812.904860] LustreError: 302111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10812.917938] LustreError: 302111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10816.678653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10817.159878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10822.212930] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10829.756689] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10835.522873] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10842.576343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10842.865610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10847.979582] LustreError: 303940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10848.935433] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10853.363614] LustreError: 304242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10857.136487] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10858.476710] LustreError: 303941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10862.568555] LustreError: 303942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10862.601211] LustreError: 303942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10872.815171] LustreError: 304242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10872.838374] LustreError: 304242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10878.692138] loop1: detected capacity change from 0 to 204800000 [10878.979458] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10878.991785] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10879.013923] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10884.117987] loop1: detected capacity change from 0 to 204800000 [10889.188955] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10889.326800] Lustre: fs6589-OST0000: new disk, initializing [10889.330349] Lustre: srv-fs6589-OST0000: No data found on store. Initialize space. [10889.339512] Lustre: 305845:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10889.428705] Lustre: fs6589-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10890.505918] LustreError: 304242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10890.524928] LustreError: 304242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10896.262914] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10905.513083] Lustre: server umount fs6589-OST0000 complete [10920.530541] loop1: detected capacity change from 0 to 204800000 [10924.007048] LustreError: 303941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10924.027014] LustreError: 303941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [10926.049986] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10926.367940] Lustre: fs6589-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10929.130897] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10932.210369] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10940.802809] loop2: detected capacity change from 0 to 204800000 [10940.975868] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10940.987209] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10940.995774] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10944.494909] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10945.994586] loop2: detected capacity change from 0 to 204800000 [10951.022383] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10951.090320] LustreError: fs6589-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10951.098709] LustreError: fs6589-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10951.105887] LustreError: 308887:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10951.110500] LustreError: 308887:0:(tgt_mount.c:2132:server_put_super()) no obd fs6589-OST0001 [10951.114084] LustreError: 308887:0:(tgt_mount.c:116:server_deregister_mount()) fs6589-OST0001 not registered [10951.146365] Lustre: server umount fs6589-OST0001 complete [10951.149731] LustreError: 308887:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10961.618429] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10961.809873] Lustre: fs6589-OST0001: new disk, initializing [10961.818088] Lustre: srv-fs6589-OST0001: No data found on store. Initialize space. [10961.826356] Lustre: 309437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10961.923174] Lustre: fs6589-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10968.006600] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [10975.666976] Lustre: server umount fs6589-OST0000 complete [10984.397737] Lustre: server umount fs6589-OST0001 complete [10990.570202] LustreError: 304242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.586067] LustreError: 304242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10996.012490] loop1: detected capacity change from 0 to 204800000 [11001.242408] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11001.545857] Lustre: fs6589-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11008.319868] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11015.235544] loop2: detected capacity change from 0 to 204800000 [11021.265117] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11028.337596] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11037.337084] Lustre: server umount fs6589-OST0000 complete [11047.414446] Lustre: server umount fs6589-OST0001 complete [11062.256431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11062.257020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11062.288533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11062.303755] Lustre: Skipped 1 previous similar message [11067.579938] Lustre: server umount lustre-OST0000 complete [11072.481256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11072.482940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11072.488850] Lustre: Skipped 2 previous similar messages [11077.604199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11077.614995] Lustre: Skipped 2 previous similar messages [11078.055900] Lustre: server umount lustre-MDT0000 complete [11081.940529] LustreError: 302091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449112 with bad export cookie 2826322033371393582 [11081.942813] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11081.949983] LustreError: 302091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11082.389826] Lustre: server umount lustre-MDT0001 complete [11095.053720] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11098.629119] Key type lgssc unregistered [11099.060849] LNet: 315359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11099.064912] LNetError: 315359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11100.134465] LNet: Removed LNI 192.168.206.157@tcp [11101.359136] Key type .llcrypt unregistered [11101.361637] Key type ._llcrypt unregistered [11136.443644] Key type ._llcrypt registered [11136.444736] Key type .llcrypt registered [11136.526489] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [11150.872362] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11151.883832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11152.035480] alg: No test for adler32 (adler32-zlib) [11153.096168] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11153.372266] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [11155.079160] Key type lgssc registered [11156.243034] Lustre: Echo OBD driver; http://www.lustre.org/ [11190.654050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11190.671088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11191.938772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11191.989088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11192.086288] Lustre: lustre-MDT0000: new disk, initializing [11192.150537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11192.163173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11196.508723] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11206.584375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11206.690666] Lustre: 319215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11206.755783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11206.764540] Lustre: Skipped 1 previous similar message [11206.847494] Lustre: lustre-MDT0001: new disk, initializing [11206.954110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11207.007084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11207.026894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11211.534486] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11220.405400] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11225.060818] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11231.869607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11232.115729] Lustre: lustre-OST0000: new disk, initializing [11232.119231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11232.124851] Lustre: 321184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11232.185267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11238.611650] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11239.996311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11240.008809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11240.152504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11246.992365] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11249.487615] Lustre: Mounted lustre-client - version 2.17.58_2_gee80f9b [11250.261224] Lustre: Unmounted lustre-client [11259.889766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11259.897729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11259.909669] Lustre: Skipped 1 previous similar message [11259.925056] Lustre: Skipped 1 previous similar message [11265.005277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11265.015682] Lustre: Skipped 1 previous similar message [11268.074223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11268.095880] Lustre: Skipped 1 previous similar message [11273.188622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11273.204270] Lustre: Skipped 1 previous similar message [11274.207900] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11274.342845] Lustre: server umount lustre-OST0000 complete [11278.819510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11278.838055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11278.868542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11283.444227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11288.553018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11288.563318] Lustre: Skipped 3 previous similar messages [11293.151349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11293.336870] Lustre: server umount lustre-MDT0000 complete [11296.892313] LustreError: 319205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449327 with bad export cookie 5127943308503134770 [11296.894785] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11296.909272] LustreError: 319205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11297.248931] Lustre: server umount lustre-MDT0001 complete [11306.952897] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11309.027510] LNet: 1 local NIs in recovery (showing 1): 192.168.206.157@tcp [11309.392539] Key type lgssc unregistered [11309.658439] LNet: 323346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11309.662624] LNetError: 323346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11380.325903] LNet: Removed LNI 192.168.206.157@tcp [11380.965416] Key type .llcrypt unregistered [11380.967250] Key type ._llcrypt unregistered [11398.549037] Key type ._llcrypt registered [11398.551633] Key type .llcrypt registered [11398.652280] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [11415.552509] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11417.110907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11417.134245] alg: No test for adler32 (adler32-zlib) [11418.213292] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11418.465486] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [11420.127164] Key type lgssc registered [11421.329035] Lustre: Echo OBD driver; http://www.lustre.org/ [11451.580966] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:31:20 (1788449480) [11460.412700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11460.438157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11461.842502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11461.928486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11462.015843] Lustre: lustre-MDT0000: new disk, initializing [11462.155621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11462.168941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11467.170882] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11477.611520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11477.850455] Lustre: 326562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11477.917677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11477.925939] Lustre: Skipped 1 previous similar message [11478.031603] Lustre: lustre-MDT0001: new disk, initializing [11478.119327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11478.173527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11478.186503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11482.241650] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11491.724397] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11497.509806] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11504.438560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11504.699288] Lustre: lustre-OST0000: new disk, initializing [11504.702800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11504.710449] Lustre: 328533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11504.805177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11510.628510] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11513.934303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11513.948965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11514.034157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11519.708236] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11539.944208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.945521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11539.969760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11541.473350] Lustre: server umount lustre-OST0000 complete [11549.671841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11549.685888] LustreError: Skipped 1 previous similar message [11549.691123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.704747] Lustre: Skipped 1 previous similar message [11549.716967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11549.724949] Lustre: Skipped 1 previous similar message [11551.866319] Lustre: server umount lustre-MDT0000 complete [11555.686031] LustreError: 326556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449586 with bad export cookie 2710137024282771377 [11555.687456] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11555.702147] LustreError: 326556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11556.186549] Lustre: server umount lustre-MDT0001 complete [11567.621407] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11570.667273] Key type lgssc unregistered [11571.183651] LNet: 330440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11571.214111] LNetError: 330440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11571.249804] LNet: Removed LNI 192.168.206.157@tcp [11572.557290] Key type .llcrypt unregistered [11572.571631] Key type ._llcrypt unregistered [11596.881621] Key type ._llcrypt registered [11596.884492] Key type .llcrypt registered [11596.991364] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11598.208173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11598.249750] alg: No test for adler32 (adler32-zlib) [11599.266532] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11599.476743] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [11601.119235] Key type lgssc registered [11602.167773] Lustre: Echo OBD driver; http://www.lustre.org/ [11610.005932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11610.019895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11611.492834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11616.112521] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11621.861264] LustreError: 331711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11621.889467] LustreError: 331711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11623.985483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11624.269909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11628.550347] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11635.500446] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11640.913558] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11647.115089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11647.568817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11654.128738] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11661.033959] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11671.146133] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11672.831272] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11677.668144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11677.680581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11677.681594] Lustre: Skipped 2 previous similar messages [11677.699380] Lustre: Skipped 2 previous similar messages [11679.035992] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.57@tcp (stopping) [11682.785335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11682.795265] Lustre: Skipped 2 previous similar messages [11683.108059] Lustre: server umount lustre-MDT0000 complete [11684.100496] LustreError: 331708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.856801] LustreError: 331692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449717 with bad export cookie 17098481510008063040 [11686.870168] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11686.871911] LustreError: 331692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11687.906746] LustreError: 333017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11687.906968] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11687.955412] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11691.948949] LustreError: 333017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11691.978903] LustreError: 333017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11694.336428] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.57@tcp (stopping) [11694.346030] Lustre: Skipped 2 previous similar messages [11696.039688] LustreError: 331708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11696.065866] LustreError: 331708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11701.215553] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11701.466466] Lustre: server umount lustre-MDT0001 complete [11705.532826] Lustre: server umount lustre-OST0000 complete [11713.329448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11716.983136] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11719.821326] Lustre: 335889:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.157@tcp [11722.650879] Lustre: server umount MGS complete [11731.021863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11731.823746] LustreError: 336461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11731.839089] LustreError: 336461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11731.883590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11737.360564] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11745.721679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11746.086760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11750.764424] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11760.278039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11760.712313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11766.826161] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11773.670041] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11775.036454] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11779.748057] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11781.072322] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11786.724357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.725289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11786.728718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11786.728726] Lustre: Skipped 4 previous similar messages [11786.732457] Lustre: Skipped 1 previous similar message [11786.746280] LustreError: Skipped 1 previous similar message [11790.996935] Lustre: server umount lustre-OST0000 complete [11796.960945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11796.986186] Lustre: Skipped 1 previous similar message [11800.594663] Lustre: server umount lustre-MDT0000 complete [11802.083458] LustreError: 336462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11802.096982] LustreError: 336462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11804.393326] LustreError: 336443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449835 with bad export cookie 17098481510008063880 [11804.398290] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11804.405715] LustreError: 336443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11804.713565] Lustre: server umount lustre-MDT0001 complete [11815.109406] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [11817.807637] Key type lgssc unregistered [11818.087418] LNet: 339900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11818.099129] LNetError: 339900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11818.126057] LNet: Removed LNI 192.168.206.157@tcp [11819.016727] Key type .llcrypt unregistered [11819.020034] Key type ._llcrypt unregistered [11837.766560] Key type ._llcrypt registered [11837.770372] Key type .llcrypt registered [11837.993828] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [11854.128055] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [11855.734466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11855.888026] alg: No test for adler32 (adler32-zlib) [11857.140345] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11857.468717] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [11859.127585] Key type lgssc registered [11860.395457] Lustre: Echo OBD driver; http://www.lustre.org/ [11900.150437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11900.169924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11901.458387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11901.487369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11901.618178] Lustre: lustre-MDT0000: new disk, initializing [11901.780457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11901.811470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11906.670129] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11918.202369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11918.329496] Lustre: 343061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11918.426695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11918.439420] Lustre: Skipped 1 previous similar message [11918.576538] Lustre: lustre-MDT0001: new disk, initializing [11918.738154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11918.790109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11918.808448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11923.727273] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11934.301778] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11940.239945] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11947.449322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11947.829762] Lustre: lustre-OST0000: new disk, initializing [11947.839303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11947.853990] Lustre: 345028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11947.981115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11953.713959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11953.727350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11953.839481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11954.395811] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [11964.850420] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11969.857357] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11970.176836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11975.999457] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11976.306813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11979.744661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11979.749947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11979.759144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11979.772242] Lustre: Skipped 1 previous similar message [11984.867266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11984.888079] Lustre: Skipped 2 previous similar messages [11989.995878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11990.007525] Lustre: Skipped 1 previous similar message [11993.055121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11993.409534] Lustre: server umount lustre-OST0000 complete [12000.224909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12000.236217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12000.239282] Lustre: Skipped 1 previous similar message [12000.259114] Lustre: Skipped 1 previous similar message [12003.644905] Lustre: server umount lustre-MDT0000 complete [12007.777570] LustreError: 343052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450038 with bad export cookie 4620742927227053002 [12007.783112] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12007.788893] LustreError: 343052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12008.216520] Lustre: server umount lustre-MDT0001 complete [12019.421894] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:40:48 (1788450048) [12035.390567] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [12044.608586] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [12084.341423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.686695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12084.741315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12084.868725] Lustre: lustre-MDT0000: new disk, initializing [12084.989624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12085.023581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12089.710306] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12100.963532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12101.164098] Lustre: 349792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12101.243092] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12101.247299] Lustre: Skipped 1 previous similar message [12101.328658] Lustre: lustre-MDT0001: new disk, initializing [12101.437257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12101.477128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12101.491694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12105.762783] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12115.103802] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12121.879886] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12129.933807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12130.266798] Lustre: lustre-OST0000: new disk, initializing [12130.272795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12130.282491] Lustre: 351759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12130.381315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12136.487961] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12137.554492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12137.571981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12137.674747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12145.221667] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12178.945262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12178.945996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12178.959864] LustreError: Skipped 2 previous similar messages [12178.962786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12178.985555] Lustre: Skipped 1 previous similar message [12183.923511] Lustre: server umount lustre-OST0000 complete [12188.643802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12188.656779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12188.693121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12188.705086] Lustre: Skipped 1 previous similar message [12194.882809] Lustre: server umount lustre-MDT0000 complete [12198.781608] LustreError: 349783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450229 with bad export cookie 4620742927227053884 [12198.790834] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12198.791880] LustreError: 349783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12199.075991] Lustre: server umount lustre-MDT0001 complete [12209.712966] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12212.780301] Key type lgssc unregistered [12213.132774] LNet: 353914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12213.142120] LNetError: 353914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12214.190165] LNet: Removed LNI 192.168.206.157@tcp [12215.280949] Key type .llcrypt unregistered [12215.284104] Key type ._llcrypt unregistered [12237.246334] Key type ._llcrypt registered [12237.248993] Key type .llcrypt registered [12237.335949] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [12238.438119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12238.502616] alg: No test for adler32 (adler32-zlib) [12239.582751] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12239.780190] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [12241.440346] Key type lgssc registered [12242.485741] Lustre: Echo OBD driver; http://www.lustre.org/ [12250.584519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12250.602736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12252.182578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12258.346358] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12262.384137] LustreError: 355184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12262.396685] LustreError: 355184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12267.505831] LustreError: 355185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.778543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.094269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12273.883311] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12281.954110] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12288.041450] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12295.409078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12295.762977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12301.957329] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12309.390706] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12323.812831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12323.816363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.819050] LustreError: Skipped 1 previous similar message [12323.834813] Lustre: Skipped 1 previous similar message [12323.837936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12323.844887] Lustre: Skipped 1 previous similar message [12328.944377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12329.526858] Lustre: server umount lustre-OST0000 complete [12334.056434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12334.059429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12334.076405] Lustre: Skipped 1 previous similar message [12334.079280] Lustre: Skipped 1 previous similar message [12339.172820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12339.179823] Lustre: Skipped 2 previous similar messages [12339.559326] Lustre: server umount lustre-MDT0000 complete [12343.657373] LustreError: 355164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450374 with bad export cookie 1325371385016504868 [12343.670992] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12343.675214] LustreError: 355164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12344.268512] Lustre: server umount lustre-MDT0001 complete [12354.248190] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12357.453303] Key type lgssc unregistered [12357.814502] LNet: 360690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12357.823748] LNetError: 360690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12357.842073] LNet: Removed LNI 192.168.206.157@tcp [12358.756524] Key type .llcrypt unregistered [12358.761911] Key type ._llcrypt unregistered [12374.779079] Key type ._llcrypt registered [12374.780496] Key type .llcrypt registered [12374.894278] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [12376.375147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12376.417658] alg: No test for adler32 (adler32-zlib) [12377.665832] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12377.998138] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [12379.711119] Key type lgssc registered [12381.567327] Lustre: Echo OBD driver; http://www.lustre.org/ [12390.555456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12390.603896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12392.251337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12397.184487] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12402.667227] LustreError: 364429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12402.688109] LustreError: 364429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12407.327967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12407.923352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12413.607969] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12421.882905] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12428.370145] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12435.669814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12436.224430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12443.300015] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12452.115900] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12463.071873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12463.073621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12463.090293] LustreError: Skipped 1 previous similar message [12463.111227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12463.117743] Lustre: Skipped 1 previous similar message [12463.136640] Lustre: Skipped 1 previous similar message [12467.319045] Lustre: server umount lustre-OST0000 complete [12473.326777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12473.345202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12473.351960] Lustre: Skipped 1 previous similar message [12477.856617] Lustre: server umount lustre-MDT0000 complete [12478.433906] LustreError: 364424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12478.460905] LustreError: 364424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12481.554055] LustreError: 364346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450512 with bad export cookie 7407951836362238970 [12481.569266] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12482.099604] Lustre: server umount lustre-MDT0001 complete [12494.633594] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12498.593701] Key type lgssc unregistered [12499.093760] LNet: 371981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12499.109336] LNetError: 371981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12499.124562] LNet: Removed LNI 192.168.206.157@tcp [12500.386745] Key type .llcrypt unregistered [12500.389635] Key type ._llcrypt unregistered [12521.338110] Key type ._llcrypt registered [12521.347350] Key type .llcrypt registered [12521.544365] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [12523.004731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12523.214614] alg: No test for adler32 (adler32-zlib) [12524.463564] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12524.904163] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [12526.615320] Key type lgssc registered [12528.154182] Lustre: Echo OBD driver; http://www.lustre.org/ [12538.022551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12538.059096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12539.636333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12544.918687] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12548.079543] LustreError: 375747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.102678] LustreError: 375747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12549.089599] LustreError: 375746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12554.224808] LustreError: 375747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12554.418353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12554.821318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12560.051193] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12566.860776] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12572.595517] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12579.433832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12579.972214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12586.998881] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12595.234942] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12610.026141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12610.038405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12610.048165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12610.060169] Lustre: Skipped 1 previous similar message [12612.824859] Lustre: server umount lustre-OST0000 complete [12620.263703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12620.268290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12620.274059] Lustre: Skipped 2 previous similar messages [12620.295728] Lustre: Skipped 1 previous similar message [12623.423701] Lustre: server umount lustre-MDT0000 complete [12625.386071] LustreError: 375740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12625.420937] LustreError: 375740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12627.277988] LustreError: 375676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450658 with bad export cookie 5089299944291112128 [12627.283193] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12627.290161] LustreError: 375676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12627.745842] Lustre: server umount lustre-MDT0001 complete [12640.125758] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12643.481418] Key type lgssc unregistered [12643.898685] LNet: 383435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12643.932852] LNetError: 383435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12644.977382] LNet: Removed LNI 192.168.206.157@tcp [12646.116165] Key type .llcrypt unregistered [12646.119764] Key type ._llcrypt unregistered [12664.426021] Key type ._llcrypt registered [12664.429364] Key type .llcrypt registered [12664.578804] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [12665.726729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12665.757605] alg: No test for adler32 (adler32-zlib) [12666.760976] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12667.027680] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [12668.771280] Key type lgssc registered [12670.094567] Lustre: Echo OBD driver; http://www.lustre.org/ [12679.523680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12679.558700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12681.492061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12687.532104] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12691.956836] LustreError: 387476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12691.987900] LustreError: 387476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12697.077429] LustreError: 387475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12697.901217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12698.290712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12703.081538] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12712.312767] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12718.423720] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12725.481032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12725.928392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12732.277370] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [12740.118059] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12753.890270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12753.896291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12753.915872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12753.926884] Lustre: Skipped 1 previous similar message [12759.010062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12759.019580] Lustre: Skipped 1 previous similar message [12759.711825] Lustre: server umount lustre-OST0000 complete [12764.141034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12764.150954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12764.164573] Lustre: Skipped 2 previous similar messages [12764.176345] Lustre: Skipped 1 previous similar message [12769.253176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12769.262848] Lustre: Skipped 1 previous similar message [12770.388438] Lustre: server umount lustre-MDT0000 complete [12774.304575] LustreError: 387420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450805 with bad export cookie 2160613442486980206 [12774.318601] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12774.322601] LustreError: 387420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12774.662229] Lustre: server umount lustre-MDT0001 complete [12786.661840] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12789.741450] Key type lgssc unregistered [12790.116498] LNet: 393428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12790.123514] LNetError: 393428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12790.144766] LNet: Removed LNI 192.168.206.157@tcp [12791.221466] Key type .llcrypt unregistered [12791.227437] Key type ._llcrypt unregistered [12894.157387] Key type ._llcrypt registered [12894.161755] Key type .llcrypt registered [12894.260807] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12894.547462] Key type .llcrypt unregistered [12894.552063] Key type ._llcrypt unregistered [12975.182599] Key type ._llcrypt registered [12975.189366] Key type .llcrypt registered [12975.328132] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [12975.644439] Key type .llcrypt unregistered [12975.648567] Key type ._llcrypt unregistered [13005.846079] Key type ._llcrypt registered [13005.850022] Key type .llcrypt registered [13005.964486] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13006.277288] Key type .llcrypt unregistered [13006.282177] Key type ._llcrypt unregistered [13039.879481] Key type ._llcrypt registered [13039.883382] Key type .llcrypt registered [13039.988090] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13040.326323] Key type .llcrypt unregistered [13040.334884] Key type ._llcrypt unregistered [13122.629154] Key type ._llcrypt registered [13122.631344] Key type .llcrypt registered [13122.763656] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13123.233500] Key type .llcrypt unregistered [13123.237692] Key type ._llcrypt unregistered [13154.505377] Key type ._llcrypt registered [13154.507663] Key type .llcrypt registered [13154.565315] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13154.867447] Key type .llcrypt unregistered [13154.870587] Key type ._llcrypt unregistered [13187.870748] Key type ._llcrypt registered [13187.873373] Key type .llcrypt registered [13187.970942] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13188.238896] Key type .llcrypt unregistered [13188.244340] Key type ._llcrypt unregistered [13217.291455] Key type ._llcrypt registered [13217.293865] Key type .llcrypt registered [13217.425921] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13217.737295] Key type .llcrypt unregistered [13217.740492] Key type ._llcrypt unregistered [13247.702788] Key type ._llcrypt registered [13247.704574] Key type .llcrypt registered [13247.834882] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13248.104845] Key type .llcrypt unregistered [13248.111681] Key type ._llcrypt unregistered [13276.048678] Key type ._llcrypt registered [13276.050923] Key type .llcrypt registered [13276.148923] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13276.405367] Key type .llcrypt unregistered [13276.410320] Key type ._llcrypt unregistered [13340.817588] Key type ._llcrypt registered [13340.819626] Key type .llcrypt registered [13340.874928] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13341.117385] Key type .llcrypt unregistered [13341.122681] Key type ._llcrypt unregistered [13374.315548] Key type ._llcrypt registered [13374.319047] Key type .llcrypt registered [13374.461445] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13374.849505] Key type .llcrypt unregistered [13374.857093] Key type ._llcrypt unregistered [13403.634815] Key type ._llcrypt registered [13403.637889] Key type .llcrypt registered [13403.739423] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13404.072045] Key type .llcrypt unregistered [13404.074886] Key type ._llcrypt unregistered [13456.491897] Key type ._llcrypt registered [13456.493207] Key type .llcrypt registered [13456.603642] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [13474.667501] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [13475.990852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13476.049131] alg: No test for adler32 (adler32-zlib) [13477.106708] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13477.331609] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [13479.032545] Key type lgssc registered [13480.265646] Lustre: Echo OBD driver; http://www.lustre.org/ [13517.789921] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13517.834450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13519.127542] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13519.159854] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13519.255910] Lustre: tst250fs-MDT0000: new disk, initializing [13519.341850] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13519.356625] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13524.067698] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [13534.793957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13534.929330] Lustre: 413996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13534.990931] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13534.995138] Lustre: Skipped 1 previous similar message [13535.094383] Lustre: tst250fs-MDT0001: new disk, initializing [13535.192696] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13535.240270] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13535.256601] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13540.073872] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [13544.631306] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13550.913380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13551.138293] Lustre: tst250fs-OST0000: new disk, initializing [13551.141176] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13551.147425] Lustre: 415629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13551.231667] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13552.199700] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13552.205512] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13552.346591] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13556.577201] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [13566.271417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13566.376720] Lustre: tst250fs-OST0001: new disk, initializing [13566.379862] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13566.384464] Lustre: 416497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13566.471435] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13572.226203] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13572.233828] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13572.370728] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13572.866863] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [13578.257704] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13580.595716] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13580.601985] Lustre: Skipped 2 previous similar messages [13601.761884] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13601.768865] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13601.785377] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13603.300754] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13603.303830] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13603.313075] Lustre: Skipped 1 previous similar message [13604.898546] Lustre: server umount tst250fs-MDT0000 complete [13608.771495] LustreError: 413986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451639 with bad export cookie 9163399022548303433 [13608.781794] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13608.792135] LustreError: 413986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13609.236765] Lustre: server umount tst250fs-MDT0001 complete [13622.947046] Lustre: server umount tst250fs-OST0000 complete [13636.266183] Lustre: server umount tst250fs-OST0001 complete [13655.859375] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [13664.569603] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [13696.410923] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:08:45 (1788451725) [13697.908879] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13715.335371] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [13724.598814] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [13754.041953] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13504 sec ========= 12:09:43 (1788451783) [13755.792637] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:09:45 (1788451785) === [13759.214618] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:09:48 (1788451788) === [13797.017772] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [13799.724784] Key type lgssc unregistered [13800.113045] LNet: 425644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13800.123097] LNetError: 425644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13800.154462] LNet: Removed LNI 192.168.206.157@tcp [13801.022209] Key type .llcrypt unregistered [13801.027727] Key type ._llcrypt unregistered