[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 618605654 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002565] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007439] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010112] LSM: Security Framework initializing [ 0.011040] Yama: becoming mindful. [ 0.012024] SELinux: Initializing. [ 0.013048] *** VALIDATE selinux *** [ 0.020351] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024589] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025125] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027073] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028086] *** VALIDATE tmpfs *** [ 0.029364] *** VALIDATE proc *** [ 0.030180] *** VALIDATE cgroup *** [ 0.031006] *** VALIDATE cgroup2 *** [ 0.033128] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034129] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036022] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.039784] debug: unmapping init [mem 0xffffffffb0e59000-0xffffffffb0e60fff] [ 0.041896] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042583] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043017] ... version: 2 [ 0.044008] ... bit width: 48 [ 0.045007] ... generic registers: 4 [ 0.046000] ... value mask: 0000ffffffffffff [ 0.047010] ... max period: 00007fffffffffff [ 0.048008] ... fixed-purpose events: 3 [ 0.049006] ... event mask: 000000070000000f [ 0.050255] rcu: Hierarchical SRCU implementation. [ 0.052324] smp: Bringing up secondary CPUs ... [ 0.053533] x86: Booting SMP configuration: [ 0.054024] .... node #0, CPUs: #1 #2 #3 [ 0.057105] smp: Brought up 1 node, 4 CPUs [ 0.059009] smpboot: Max logical packages: 1 [ 0.060013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131016] node 0 deferred pages initialised in 68ms [ 0.134142] devtmpfs: initialized [ 0.135225] x86/mm: Memory block size: 128MB [ 0.139125] gcov: version magic: 0x41383552 [ 0.141253] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142063] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144264] pinctrl core: initialized pinctrl subsystem [ 0.146132] [ 0.146633] ************************************************************* [ 0.148009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150009] ** ** [ 0.152009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154009] ** ** [ 0.157008] ** This means that this kernel is built to expose internal ** [ 0.159007] ** IOMMU data structures, which may compromise security on ** [ 0.161009] ** your system. ** [ 0.163007] ** ** [ 0.165008] ** If you see this message and you are not debugging the ** [ 0.167009] ** kernel, report this immediately to your vendor! ** [ 0.169009] ** ** [ 0.171009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173013] ************************************************************* [ 0.175593] NET: Registered protocol family 16 [ 0.177377] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186031] cpuidle: using governor menu [ 0.189167] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191346] PCI: Using configuration type 1 for base access [ 0.192132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202070] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.210034] cryptd: max_cpu_qlen set to 1000 [ 0.214217] ACPI: Added _OSI(Module Device) [ 0.215010] ACPI: Added _OSI(Processor Device) [ 0.217020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222964] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229553] ACPI: Interpreter enabled [ 0.230044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.232011] ACPI: Using IOAPIC for interrupt routing [ 0.234081] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237333] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246999] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.249031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.252013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.255064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259197] acpiphp: Slot [2] registered [ 0.261084] acpiphp: Slot [5] registered [ 0.262145] acpiphp: Slot [6] registered [ 0.263079] acpiphp: Slot [7] registered [ 0.265091] acpiphp: Slot [8] registered [ 0.266111] acpiphp: Slot [9] registered [ 0.268116] acpiphp: Slot [10] registered [ 0.270091] acpiphp: Slot [3] registered [ 0.272152] acpiphp: Slot [4] registered [ 0.273077] acpiphp: Slot [11] registered [ 0.275127] acpiphp: Slot [12] registered [ 0.277094] acpiphp: Slot [13] registered [ 0.278101] acpiphp: Slot [14] registered [ 0.280076] acpiphp: Slot [15] registered [ 0.282076] acpiphp: Slot [16] registered [ 0.283072] acpiphp: Slot [17] registered [ 0.285147] acpiphp: Slot [18] registered [ 0.287086] acpiphp: Slot [19] registered [ 0.289075] acpiphp: Slot [20] registered [ 0.291077] acpiphp: Slot [21] registered [ 0.292081] acpiphp: Slot [22] registered [ 0.294155] acpiphp: Slot [23] registered [ 0.296081] acpiphp: Slot [24] registered [ 0.298071] acpiphp: Slot [25] registered [ 0.299080] acpiphp: Slot [26] registered [ 0.301138] acpiphp: Slot [27] registered [ 0.303135] acpiphp: Slot [28] registered [ 0.305094] acpiphp: Slot [29] registered [ 0.306073] acpiphp: Slot [30] registered [ 0.307083] acpiphp: Slot [31] registered [ 0.308064] PCI host bridge to bus 0000:00 [ 0.310017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.316017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.319071] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.322020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.325020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.327150] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.331048] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.334237] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344821] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.351055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.354013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.359014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362593] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365748] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.368037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.371798] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.398051] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405276] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.412020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.419023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.434017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.444892] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.466015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.489057] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.515017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.540490] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.547012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.555013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.571012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.585930] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.593011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.610012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.662012] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.689787] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.698012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.708012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.733012] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.746606] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.768012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.793013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.839013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.864662] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.867336] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.869276] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.871338] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.873174] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.878098] iommu: Default domain type: Passthrough [ 0.880337] SCSI subsystem initialized [ 0.881101] ACPI: bus type USB registered [ 0.883086] usbcore: registered new interface driver usbfs [ 0.884040] usbcore: registered new interface driver hub [ 0.886052] usbcore: registered new device driver usb [ 0.888189] pps_core: LinuxPPS API ver. 1 registered [ 0.889007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.892062] PTP clock support registered [ 0.894108] EDAC MC: Ver: 3.0.0 [ 0.897099] PCI: Using ACPI for IRQ routing [ 0.898759] NetLabel: Initializing [ 0.899006] NetLabel: domain hash size = 128 [ 0.900006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.901098] NetLabel: unlabeled traffic allowed by default [ 0.903038] vgaarb: loaded [ 0.905346] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.906010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.914915] clocksource: Switched to clocksource kvm-clock [ 1.018047] VFS: Disk quotas dquot_6.6.0 [ 1.019255] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.021223] *** VALIDATE ramfs *** [ 1.022197] *** VALIDATE hugetlbfs *** [ 1.023388] pnp: PnP ACPI init [ 1.025343] pnp: PnP ACPI: found 6 devices [ 1.114677] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.117703] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.119588] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.121438] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.123542] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.125620] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.128187] NET: Registered protocol family 2 [ 1.130260] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.134391] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.137432] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.142268] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.145372] TCP: Hash tables configured (established 65536 bind 65536) [ 1.148443] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.151083] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.153446] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.155983] NET: Registered protocol family 1 [ 1.159160] RPC: Registered named UNIX socket transport module. [ 1.161111] RPC: Registered udp transport module. [ 1.162479] RPC: Registered tcp transport module. [ 1.163861] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.165701] NET: Registered protocol family 44 [ 1.167190] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.169089] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.170786] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.172993] PCI: CLS 0 bytes, default 64 [ 1.174965] Unpacking initramfs... [ 2.469263] debug: unmapping init [mem 0xffff95babcc54000-0xffff95babffbffff] [ 2.472713] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.474535] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.476881] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.968629] Initialise system trusted keyrings [ 2.970498] Key type blacklist registered [ 2.973096] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.984052] zbud: loaded [ 2.988196] *** VALIDATE nfs *** [ 2.989589] *** VALIDATE nfs4 *** [ 2.991265] pstore: using deflate compression [ 2.996423] Platform Keyring initialized [ 3.143248] NET: Registered protocol family 38 [ 3.144881] Key type asymmetric registered [ 3.146403] Asymmetric key parser 'x509' registered [ 3.147805] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.151133] io scheduler mq-deadline registered [ 3.152725] io scheduler kyber registered [ 3.155199] io scheduler bfq registered [ 3.158526] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.161487] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.164697] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.167835] ACPI: Power Button [PWRF] [ 3.173253] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.179209] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.239241] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.249493] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.269486] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.299459] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.326730] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.332778] Non-volatile memory driver v1.3 [ 3.334205] Linux agpgart interface v0.103 [ 3.364846] virtio_blk virtio1: [vda] 145952 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.367357] vda: detected capacity change from 0 to 74727424 [ 3.380483] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.383022] vdb: detected capacity change from 0 to 1073741824 [ 3.395533] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.398152] vdc: detected capacity change from 0 to 2621440000 [ 3.425433] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.427842] vdd: detected capacity change from 0 to 2621440000 [ 3.453136] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.456347] vde: detected capacity change from 0 to 4294967296 [ 3.480845] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.483389] vdf: detected capacity change from 0 to 4294967296 [ 3.489178] libphy: Fixed MDIO Bus: probed [ 3.495121] usbcore: registered new interface driver usbserial_generic [ 3.497049] usbserial: USB Serial support registered for generic [ 3.498668] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.502108] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.503462] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.505861] mousedev: PS/2 mouse device common for all mice [ 3.508173] rtc_cmos 00:05: RTC can wake from S4 [ 3.511711] rtc_cmos 00:05: registered as rtc0 [ 3.513076] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.516141] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.517129] intel_pstate: CPU model not supported [ 3.522477] hid: raw HID events driver (C) Jiri Kosina [ 3.524902] usbcore: registered new interface driver usbhid [ 3.525257] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.526891] usbhid: USB HID core driver [ 3.527035] drop_monitor: Initializing network drop monitor service [ 3.532951] Initializing XFRM netlink socket [ 3.534527] NET: Registered protocol family 10 [ 3.536174] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.537108] Segment Routing with IPv6 [ 3.540014] NET: Registered protocol family 17 [ 3.542099] mpls_gso: MPLS GSO support [ 3.546550] RAS: Correctable Errors collector initialized. [ 3.549799] AVX version of gcm_enc/dec engaged. [ 3.551138] AES CTR mode by8 optimization enabled [ 3.631286] sched_clock: Marking stable (3631248851, 0)->(4406531628, -775282777) [ 3.634160] registered taskstats version 1 [ 3.635745] Loading compiled-in X.509 certificates [ 3.637823] zswap: loaded using pool lzo/zbud [ 3.663802] Key type big_key registered [ 3.675077] Key type encrypted registered [ 3.676653] ima: No TPM chip found, activating TPM-bypass! [ 3.678177] ima: Allocated hash algorithm: sha1 [ 3.679503] ima: No architecture policies found [ 3.680866] evm: Initialising EVM extended attributes: [ 3.682312] evm: security.selinux [ 3.683275] evm: security.ima [ 3.684137] evm: security.capability [ 3.685168] evm: HMAC attrs: 0x1 [ 3.687526] rtc_cmos 00:05: setting system clock to 2026-08-06 11:41:37 UTC (1786016497) [ 3.693294] debug: unmapping init [mem 0xffffffffb1e03000-0xffffffffb1ffffff] [ 3.695714] debug: unmapping init [mem 0xffffffffb0b82000-0xffffffffb0e58fff] [ 3.704093] Write protecting the kernel read-only data: 28672k [ 3.707055] debug: unmapping init [mem 0xffffffffaf203000-0xffffffffaf3fffff] [ 3.709273] debug: unmapping init [mem 0xffffffffafb14000-0xffffffffafbfffff] [ 3.739831] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.747310] systemd[1]: Detected virtualization kvm. [ 3.748778] systemd[1]: Detected architecture x86-64. [ 3.750292] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.775835] systemd[1]: No hostname configured. [ 3.777251] systemd[1]: Set hostname to . [ 3.778865] random: systemd: uninitialized urandom read (16 bytes read) [ 3.780759] systemd[1]: Initializing machine ID from random generator. [ 3.831884] random: ln: uninitialized urandom read (6 bytes read) [ 3.899306] random: systemd: uninitialized urandom read (16 bytes read) [ 3.901498] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.905201] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.908256] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.461350] device-mapper: uevent: version 1.0.3 [ 4.463152] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.131669] virtio_net virtio0 ens2: renamed from eth0 [ 5.146920] random: fast init done [ 5.146922] random: fast init done [ 5.199830] scsi host0: ata_piix [ 5.209558] scsi host1: ata_piix [ 5.211711] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.213915] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.596678] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.208193] random: crng init done [ 10.210384] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.513819] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.644629] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.943232] SELinux: Disabled at runtime. [ 12.006155] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.014360] systemd[1]: Detected virtualization kvm. [ 12.015693] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.463481] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.465992] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.470915] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.474289] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.477036] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.483116] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.491383] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ 12.606381] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.876514] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.170076] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.209058] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.351356] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.385339] EDAC sbridge: Ver: 1.1.2 [ 15.293715] Key type dns_resolver registered [ 15.580768] NFS: Registering the id_resolver key type [ 15.582373] Key type id_resolver registered [ 15.583662] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg306-server login: [ 41.270517] libcfs: loading out-of-tree module taints kernel. [ 41.290150] Key type ._llcrypt registered [ 41.291829] Key type .llcrypt registered [ 41.342823] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 49.885001] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 50.640707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.649502] alg: No test for adler32 (adler32-zlib) [ 52.062932] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 53.747342] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 55.648356] Key type lgssc registered [ 58.212140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.905214] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 133.915276] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 139.776130] hrtimer: interrupt took 19123056 ns [ 159.156192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.218793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.909591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.989225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.229204] Lustre: lustre-MDT0000: new disk, initializing [ 161.388436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.439149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.808465] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.498978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.705616] Lustre: 6487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.761234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 188.772851] Lustre: Skipped 1 previous similar message [ 188.995717] Lustre: lustre-MDT0001: new disk, initializing [ 189.126197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 189.187291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 189.202787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.665415] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.248857] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.357728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.735197] Lustre: lustre-OST0000: new disk, initializing [ 215.737263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.748503] Lustre: 8425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.850770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.746772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 222.767647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 222.912918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 223.891691] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.173925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.383150] Lustre: lustre-OST0001: new disk, initializing [ 245.386725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.398187] Lustre: 9497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.488292] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 253.528062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 253.548184] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 253.554808] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.691954] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 267.510358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 276.020576] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 285.232851] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing check_logdir /tmp/testlogs/ [ 291.258337] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing yml_node [ 298.083510] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 301.216200] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 304.874792] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 306.479519] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 6 07:46:38 EDT 2026 [ 328.372677] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 330.473389] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 351.202669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.233381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.248851] Lustre: Skipped 2 previous similar messages [ 356.321170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 356.325835] Lustre: Skipped 6 previous similar messages [ 357.830833] Lustre: server umount lustre-MDT0000 complete [ 362.305556] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786016856 with bad export cookie 3777586960580680966 [ 362.319236] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.323079] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 366.561484] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.562029] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.585216] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 366.601381] Lustre: Skipped 1 previous similar message [ 366.613763] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.621065] Lustre: Skipped 1 previous similar message [ 367.587147] LustreError: 6494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.619513] LustreError: 6494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 369.040398] Lustre: server umount lustre-MDT0001 complete [ 374.161690] Lustre: server umount lustre-OST0000 complete [ 379.419663] Lustre: server umount lustre-OST0001 complete [ 389.086603] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 408.169778] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 466.757890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.298142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 467.410126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 467.625877] Lustre: lustre-MDT0000: new disk, initializing [ 467.843157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.904845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 476.051439] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 491.762290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.920602] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 492.076505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 492.090959] Lustre: Skipped 1 previous similar message [ 492.245168] Lustre: lustre-MDT0001: new disk, initializing [ 492.456709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 492.574124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 492.619456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 499.811651] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 513.567489] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 523.648956] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 534.724659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 535.334785] Lustre: lustre-OST0000: new disk, initializing [ 535.345398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 535.367923] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 535.503820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.499492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 536.516213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 536.773086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 544.427284] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 560.418452] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 568.698787] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 569.137081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 576.638436] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 576.991909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 582.654629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.658579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 582.670467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 582.694181] Lustre: Skipped 2 previous similar messages [ 582.730898] Lustre: Skipped 3 previous similar messages [ 587.789524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 587.810754] Lustre: Skipped 1 previous similar message [ 594.401699] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 594.690348] Lustre: server umount lustre-OST0000 complete [ 603.118867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.136751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.139203] Lustre: Skipped 1 previous similar message [ 603.163541] Lustre: Skipped 3 previous similar messages [ 607.112237] Lustre: server umount lustre-MDT0000 complete [ 608.236614] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.268486] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 612.062734] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786017105 with bad export cookie 3777586960580682499 [ 612.082100] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.440275] Lustre: server umount lustre-MDT0001 complete [ 621.001464] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 07:51:52 (1786017112) [ 645.527768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.415566] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.560653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.746961] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.202997] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 662.006672] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.032851] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 664.488471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.549106] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 679.896041] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 687.788956] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 690.502893] Lustre: Failing over lustre-MDT0000 [ 690.658595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 690.668077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.684781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 692.846618] Lustre: server umount lustre-MDT0000 complete [ 694.760504] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.791480] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 711.148268] Lustre: 3615:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786017188/real 1786017188] req@ffff95ba035a0a80 x1872774086967808/t0(0) o400->MGC192.168.203.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786017204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 711.176877] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.722791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.380063] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x346caf537d8f2889 [ 721.404868] Lustre: MGC192.168.203.106@tcp: Connection restored to 0@lo (at 0@lo) [ 721.783299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 721.789163] Lustre: Skipped 1 previous similar message [ 721.878068] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 727.037111] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 727.055600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 727.057741] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 727.193497] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 740.245175] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 747.491454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 747.501511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 747.524650] Lustre: Skipped 1 previous similar message [ 747.540594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 751.499285] Lustre: server umount lustre-MDT0000 complete [ 754.676836] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 754.706434] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 756.568907] LustreError: 21506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786017250 with bad export cookie 3777586960580683913 [ 756.570664] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 756.586725] LustreError: 21506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 757.484302] Lustre: server umount lustre-MDT0001 complete [ 768.203082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.241545] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 784.753569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.279058] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 801.894865] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 811.163128] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 816.355268] Lustre: Failing over lustre-MDT0000 [ 816.605450] Lustre: server umount lustre-MDT0000 complete [ 819.170146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.175224] LustreError: 23280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 819.183474] Lustre: Skipped 2 previous similar messages [ 819.249715] LustreError: 23280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 834.530442] Lustre: 3617:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786017312/real 1786017312] req@ffff95bb3f782300 x1872774087001600/t0(0) o400->MGC192.168.203.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786017328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 834.565076] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 839.329679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.769607] LustreError: 3614:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95bb3f782680 x1872774087003904/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 845.358653] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 850.405169] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 850.417461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 850.425131] Lustre: Skipped 1 previous similar message [ 850.438403] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 850.608326] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 861.439238] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 869.866444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 869.875064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 869.884112] Lustre: Skipped 1 previous similar message [ 869.907162] Lustre: Skipped 3 previous similar messages [ 872.868284] Lustre: server umount lustre-MDT0000 complete [ 878.388764] LustreError: 23260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786017372 with bad export cookie 3777586960580684858 [ 878.410084] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 878.411358] LustreError: 23260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 879.143455] Lustre: server umount lustre-MDT0001 complete [ 892.910200] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 07:56:24 (1786017384) [ 914.817474] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 926.670590] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 978.199498] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 991.578199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.113988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 992.180434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 992.358784] Lustre: lustre-MDT0000: new disk, initializing [ 992.473910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.477025] Lustre: Skipped 3 previous similar messages [ 992.526920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 999.780487] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1015.982647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.204924] Lustre: 29675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1016.822174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1022.576607] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1033.774573] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1041.095108] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1050.687340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1051.084099] Lustre: 31646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1052.627964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1052.699874] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1052.784851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1052.798386] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1059.560473] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1071.626826] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1078.733096] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1106.929949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.952505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1113.071865] Lustre: server umount lustre-OST0000 complete [ 1119.204631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1125.076246] Lustre: server umount lustre-MDT0000 complete [ 1127.413224] LustreError: 30543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.437207] LustreError: 30543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 1130.638808] LustreError: 29667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786017624 with bad export cookie 3777586960580685264 [ 1130.650474] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.671265] LustreError: 29667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1131.637643] Lustre: server umount lustre-MDT0001 complete [ 1147.361845] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1152.053203] Key type lgssc unregistered [ 1152.559662] LNet: 33628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.583347] LNetError: 33628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.659043] LNet: Removed LNI 192.168.203.106@tcp [ 1155.333394] Key type .llcrypt unregistered [ 1155.344267] Key type ._llcrypt unregistered [ 1190.803377] Key type ._llcrypt registered [ 1190.812352] Key type .llcrypt registered [ 1191.074330] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1194.331723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1194.369659] alg: No test for adler32 (adler32-zlib) [ 1195.547955] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 1195.917520] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1197.816293] Key type lgssc registered [ 1200.178708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1212.143634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1212.181587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.014083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.221155] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1224.181412] LustreError: 35059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1224.206240] LustreError: 35059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1229.281688] LustreError: 35058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.999873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.403076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1234.570081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.450562] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1255.443340] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1264.478956] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1274.531856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1275.248225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.385951] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1283.627371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1283.750310] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1294.202430] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1320.142415] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:03:29 (1786017809) [ 1322.901850] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1325.265555] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1336.382164] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:03:46 (1786017826) [ 1349.139080] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:04:00 (1786017840) [ 1350.722056] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1361.131373] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1361.139781] Lustre: Skipped 2 previous similar messages [ 1363.891479] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:04:14 (1786017854) [ 1366.468050] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1366.470332] Lustre: Skipped 2 previous similar messages [ 1375.488609] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1375.500487] Lustre: Skipped 1 previous similar message [ 1395.416213] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1395.423738] Lustre: Skipped 5 previous similar messages [ 1399.318312] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:04:50 (1786017890) [ 1402.893769] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1407.370137] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1420.534834] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:05:10 (1786017910) [ 1429.748377] Lustre: Setting parameter general.jobid_name=TESTNAME1 in log params [ 1429.762611] Lustre: Skipped 5 previous similar messages [ 1447.904518] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:05:38 (1786017938) [ 1473.009282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.016988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1473.017618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1473.025653] Lustre: Skipped 1 previous similar message [ 1476.071590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1476.083894] Lustre: Skipped 2 previous similar messages [ 1479.015176] Lustre: server umount lustre-OST0000 complete [ 1490.401560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1490.429866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.504423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.480024] Lustre: server umount lustre-MDT0000 complete [ 1496.546744] LustreError: 35053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.575516] LustreError: 35053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1500.754097] LustreError: 35795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786017994 with bad export cookie 4725799059401016590 [ 1500.759220] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1500.798535] LustreError: 35795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1501.868331] Lustre: server umount lustre-MDT0001 complete [ 1517.374286] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1520.800660] Key type lgssc unregistered [ 1521.268644] LNet: 42940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.272470] LNetError: 42940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.342597] LNet: Removed LNI 192.168.203.106@tcp [ 1523.498219] Key type .llcrypt unregistered [ 1523.499657] Key type ._llcrypt unregistered [ 1546.543465] Key type ._llcrypt registered [ 1546.547561] Key type .llcrypt registered [ 1546.701628] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 1570.524500] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1572.422109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1572.458648] alg: No test for adler32 (adler32-zlib) [ 1573.887163] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 1574.391469] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1576.216158] Key type lgssc registered [ 1577.897254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1630.839182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1630.877685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.174487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1632.221500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1632.378857] Lustre: lustre-MDT0000: new disk, initializing [ 1632.514349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.544060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1639.316285] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1653.699558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1653.948935] Lustre: 46101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1654.043438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1654.049869] Lustre: Skipped 1 previous similar message [ 1654.270241] Lustre: lustre-MDT0001: new disk, initializing [ 1654.460464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1654.549592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1654.572142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1661.632851] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1674.633963] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1681.945586] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1690.818699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1691.240120] Lustre: lustre-OST0000: new disk, initializing [ 1691.253173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1691.263568] Lustre: 48071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1691.424453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.356808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1698.375217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1698.563911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1699.405351] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1711.462310] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1716.813433] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1717.005396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1721.792240] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1722.064493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1724.404835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1724.407950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.447596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1729.511247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1729.522563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1729.541980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.904971] Lustre: server umount lustre-OST0000 complete [ 1736.677803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1736.688508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.713797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.724512] Lustre: Skipped 1 previous similar message [ 1739.758509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.767552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.805588] Lustre: Skipped 1 previous similar message [ 1742.684875] Lustre: server umount lustre-MDT0000 complete [ 1747.520088] LustreError: 46094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018241 with bad export cookie 7345310259066374711 [ 1747.528508] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1747.534562] LustreError: 46094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1748.220156] Lustre: server umount lustre-MDT0001 complete [ 1759.737157] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:10:50 (1786018250) [ 1770.428761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.182971] LustreError: 50532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1771.206509] LustreError: 50532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1771.306324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.630398] LustreError: 50533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1776.941545] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1781.735872] LustreError: 50532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.874864] LustreError: 50532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.910399] LustreError: 50532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1791.547838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.793503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1797.804419] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1808.879482] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1816.844356] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1825.311054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1826.000771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.996673] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1843.731552] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1849.362812] LustreError: 53067:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1850.522920] Lustre: Modifying parameter general.timeout=1 in log params [ 1851.726522] Lustre: Modifying parameter general.timeout=2 in log params [ 1854.056725] Lustre: Modifying parameter general.timeout=4 in log params [ 1854.064964] Lustre: Skipped 1 previous similar message [ 1858.241474] Lustre: Modifying parameter general.timeout=8 in log params [ 1858.243559] Lustre: Skipped 3 previous similar messages [ 1866.447925] Lustre: Modifying parameter general.timeout=15 in log params [ 1866.456299] Lustre: Skipped 6 previous similar messages [ 1882.951832] Lustre: Modifying parameter general.timeout=28 in log params [ 1882.958772] Lustre: Skipped 12 previous similar messages [ 1916.058328] Lustre: Modifying parameter general.timeout=52 in log params [ 1916.068427] Lustre: Skipped 23 previous similar messages [ 1980.097536] Lustre: Modifying parameter general.timeout=107 in log params [ 1980.100488] Lustre: Skipped 54 previous similar messages [ 2027.919376] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:15:18 (1786018518) [ 2031.635434] Lustre: 60038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 2034.481628] Lustre: 60137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 2038.321724] Lustre: 60245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 2044.780506] Lustre: 60442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 2044.795815] Lustre: 60442:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2089.266084] Lustre: 61914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 2089.273225] Lustre: 61914:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2089.302903] LustreError: 61914:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 2094.779945] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:16:26 (1786018586) [ 2113.506887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.508412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2113.539397] Lustre: Skipped 2 previous similar messages [ 2113.542214] Lustre: Skipped 1 previous similar message [ 2120.098443] Lustre: server umount lustre-MDT0000 complete [ 2123.750114] LustreError: 50528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2123.790140] LustreError: 50528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2125.466735] LustreError: 51268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018619 with bad export cookie 7345310259066375593 [ 2125.467432] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2125.486128] LustreError: 51268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2126.323855] Lustre: server umount lustre-MDT0001 complete [ 2141.112503] Lustre: server umount lustre-OST0000 complete [ 2179.611155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.265633] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2180.308974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2180.317829] Lustre: Skipped 45 previous similar messages [ 2180.872574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2186.523638] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2196.905956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.094919] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2197.148236] Lustre: 64451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2206.112905] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2221.838975] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2232.440449] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2241.903742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2242.195574] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2242.671391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2242.680962] Lustre: Skipped 1 previous similar message [ 2251.461745] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2261.995318] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2264.569322] Lustre: 67025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2274.284380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2274.287281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.299394] LustreError: Skipped 1 previous similar message [ 2274.301411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2274.301415] Lustre: Skipped 4 previous similar messages [ 2274.342863] Lustre: Skipped 1 previous similar message [ 2280.396088] Lustre: server umount lustre-OST0000 complete [ 2289.633579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2293.537291] Lustre: server umount lustre-MDT0000 complete [ 2294.774942] LustreError: 64464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.801915] LustreError: 64464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2298.699630] LustreError: 64445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018792 with bad export cookie 7345310259066379030 [ 2298.713639] LustreError: 64445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2298.715212] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.631097] Lustre: server umount lustre-MDT0001 complete [ 2320.678181] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2327.705265] Key type lgssc unregistered [ 2328.357354] LNet: 68257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.377887] LNetError: 68257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.404702] LNet: Removed LNI 192.168.203.106@tcp [ 2330.527618] Key type .llcrypt unregistered [ 2330.535538] Key type ._llcrypt unregistered [ 2372.845436] Key type ._llcrypt registered [ 2372.847837] Key type .llcrypt registered [ 2372.931451] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2374.149852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.221701] alg: No test for adler32 (adler32-zlib) [ 2375.552673] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 2376.085308] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2377.872195] Key type lgssc registered [ 2379.489320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2390.212615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2390.244348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.203936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.004505] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2401.833050] LustreError: 69675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.857058] LustreError: 69675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2406.894813] LustreError: 69674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.447061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.301825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2415.119360] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2423.660280] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2429.864793] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2439.175418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2439.817800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.607630] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2459.913956] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2469.998946] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2481.633383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2481.641271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.660200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2482.147896] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.174514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2482.177567] Lustre: Skipped 1 previous similar message [ 2487.268044] LustreError: 69675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.300796] LustreError: 69675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2489.354552] Lustre: server umount lustre-MDT0000 complete [ 2492.388584] LustreError: 70406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.408606] LustreError: 70406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2494.119571] LustreError: 69655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018987 with bad export cookie 1177289065331902744 [ 2494.124026] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.134732] LustreError: 69655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2494.760572] Lustre: server umount lustre-MDT0001 complete [ 2509.969154] Lustre: server umount lustre-OST0000 complete [ 2544.936441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.313958] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2545.371776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2545.705887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2551.545297] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2563.519863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.626764] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2563.670176] Lustre: 74548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2564.336332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2570.517465] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2581.334733] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2588.629970] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2598.100577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2598.408523] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2599.051626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2607.076822] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2617.096586] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2619.658416] Lustre: 77133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2622.402475] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2631.650625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2631.670701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.697733] Lustre: Skipped 1 previous similar message [ 2631.748632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2631.758451] Lustre: Skipped 1 previous similar message [ 2636.772836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2636.792084] Lustre: Skipped 1 previous similar message [ 2637.973458] Lustre: server umount lustre-OST0000 complete [ 2645.985639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2645.996980] LustreError: Skipped 1 previous similar message [ 2646.005837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.027682] Lustre: Skipped 1 previous similar message [ 2646.042790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.633172] Lustre: server umount lustre-MDT0000 complete [ 2652.138608] LustreError: 74559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.158044] LustreError: 74559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2657.463857] LustreError: 77600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786019151 with bad export cookie 1177289065331903591 [ 2657.474417] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2657.478074] LustreError: 77600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2658.233438] Lustre: server umount lustre-MDT0001 complete [ 2674.883716] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2678.648973] Key type lgssc unregistered [ 2678.969692] LNet: 78355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2678.985032] LNetError: 78355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2679.006701] LNet: Removed LNI 192.168.203.106@tcp [ 2680.172395] Key type .llcrypt unregistered [ 2680.176522] Key type ._llcrypt unregistered [ 2712.097860] Key type ._llcrypt registered [ 2712.101745] Key type .llcrypt registered [ 2712.296877] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2713.603569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2713.631769] alg: No test for adler32 (adler32-zlib) [ 2715.392580] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 2715.834075] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2717.736230] Key type lgssc registered [ 2719.736930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2729.375577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2729.391485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.737061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2735.706746] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2740.212162] LustreError: 79768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.247057] LustreError: 79768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2745.313795] LustreError: 79769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.022192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.035904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2754.389109] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2764.569861] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2773.243599] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2782.136211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2783.087975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.020655] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2804.537871] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2812.413329] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2813.901282] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2816.322642] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2819.236238] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2819.246256] Lustre: Skipped 1 previous similar message [ 2823.591704] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2823.604247] Lustre: Skipped 3 previous similar messages [ 2831.872064] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2831.884501] Lustre: Skipped 6 previous similar messages [ 2848.040407] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2848.045637] Lustre: Skipped 13 previous similar messages [ 2880.572955] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2880.585689] Lustre: Skipped 24 previous similar messages [ 2959.669297] Lustre: 87891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2960.888091] Lustre: Disabling parameter general.jobid_var= in log params [ 2960.897540] Lustre: Skipped 45 previous similar messages [ 2963.498907] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:30:54 (1786019454) [ 2965.711875] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2968.416640] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:30:59 (1786019459) [ 2974.690626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2974.701714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.713438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.737201] Lustre: Skipped 1 previous similar message [ 2974.763431] Lustre: Skipped 1 previous similar message [ 2979.838986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2979.853091] Lustre: Skipped 1 previous similar message [ 2981.169540] Lustre: server umount lustre-OST0000 complete [ 2988.513493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2988.526942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.537106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2990.052537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.169421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2995.172332] Lustre: Skipped 2 previous similar messages [ 3000.290870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3000.312553] Lustre: Skipped 1 previous similar message [ 3001.312144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3001.854148] Lustre: server umount lustre-MDT0000 complete [ 3004.472419] LustreError: 79768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.531530] LustreError: 79768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3009.413503] LustreError: 82559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786019503 with bad export cookie 919147644175937939 [ 3009.416310] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3009.433050] LustreError: 82559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3010.966533] Lustre: server umount lustre-MDT0001 complete [ 3029.670673] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3033.516989] Key type lgssc unregistered [ 3034.062522] LNet: 89145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3034.074318] LNetError: 89145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3035.113708] LNet: Removed LNI 192.168.203.106@tcp [ 3036.351218] Key type .llcrypt unregistered [ 3036.355209] Key type ._llcrypt unregistered [ 3043.056906] Key type ._llcrypt registered [ 3043.059319] Key type .llcrypt registered [ 3043.195403] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 3050.551951] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules [ 3053.201757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3053.272230] alg: No test for adler32 (adler32-zlib) [ 3054.777251] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 3055.354922] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3057.152393] Key type lgssc registered [ 3060.207563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3062.075596] LustreError: 90399:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 3084.001028] LustreError: 90399:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 3087.312988] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3097.145664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3097.188890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.073936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.974353] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3108.336571] LustreError: 91429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.371412] LustreError: 91429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3113.442084] LustreError: 91428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.912574] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:33:26 (1786019606) [ 3118.561883] LustreError: 91429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3122.666266] LustreError: 91460:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3122.685866] LustreError: 91460:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3122.714230] LustreError: 91460:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 3123.529075] Lustre: server umount lustre-MDT0000 complete [ 3144.005451] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3147.764306] Key type lgssc unregistered [ 3148.119630] LNet: 92954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.123226] LNetError: 92954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3149.159249] LNet: Removed LNI 192.168.203.106@tcp [ 3150.340763] Key type .llcrypt unregistered [ 3150.344877] Key type ._llcrypt unregistered [ 3174.598803] Key type ._llcrypt registered [ 3174.601541] Key type .llcrypt registered [ 3174.714657] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3175.821817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3175.848673] alg: No test for adler32 (adler32-zlib) [ 3177.272262] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 3177.737376] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3179.490653] Key type lgssc registered [ 3182.321811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3197.255831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3197.290577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.003152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.284447] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3209.185439] LustreError: 94173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.208600] LustreError: 94173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3214.305803] LustreError: 94174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.032748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.239239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3223.052952] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3234.325553] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3243.135315] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3253.232573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3254.588495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.931817] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3272.629990] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3334.586394] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:37:05 (1786019825) [ 3337.432421] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3339.760734] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:37:11 (1786019831) [ 3349.988062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3350.002909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.037107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3351.014142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3351.016379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.035879] Lustre: Skipped 1 previous similar message [ 3351.079749] Lustre: Skipped 1 previous similar message [ 3354.309407] Lustre: server umount lustre-MDT0000 complete [ 3356.140536] LustreError: 94170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3356.173379] LustreError: 94170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3360.101384] LustreError: 96030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786019853 with bad export cookie 1607514483492994207 [ 3360.106914] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3360.114761] LustreError: 96030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3360.651548] Lustre: server umount lustre-MDT0001 complete [ 3376.142580] Lustre: server umount lustre-OST0000 complete [ 3392.982553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3393.530768] LustreError: 98478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.956574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.140069] LustreError: 98479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3399.421469] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3408.362541] LustreError: 98478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.386455] LustreError: 98478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3411.037515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.025709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3417.446869] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3427.298978] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3438.195787] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3455.382469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3455.770041] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3455.785118] LustreError: 98467:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3455.842099] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3455.854725] LustreError: 100566:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3455.866594] LustreError: 100566:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3455.878248] LustreError: 100566:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3456.022826] Lustre: server umount lustre-OST0000 complete [ 3456.027226] LustreError: 100566:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3470.101140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3470.522479] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3470.536177] LustreError: 98466:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3470.586262] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3470.598548] LustreError: 101032:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3470.626388] LustreError: 101032:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3470.638597] LustreError: 101032:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3470.827935] Lustre: server umount lustre-OST0000 complete [ 3470.835095] LustreError: 101032:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3491.162061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3491.351909] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3491.359266] Lustre: Found index 0 for lustre-OST0000, updating log [ 3491.369061] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3491.493954] Lustre: lustre-OST0000: new disk, initializing [ 3491.501778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3491.513634] Lustre: 101569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3491.901815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.590816] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3505.149133] Lustre: Failing over lustre-OST0000 [ 3505.467334] Lustre: server umount lustre-OST0000 complete [ 3510.497185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.519073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3510.521442] Lustre: Skipped 1 previous similar message [ 3515.874039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3515.885253] Lustre: Skipped 1 previous similar message [ 3517.121937] Lustre: server umount lustre-MDT0000 complete [ 3521.005602] LustreError: 98479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.027494] LustreError: 98479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3521.474298] LustreError: 98458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786020015 with bad export cookie 1607514483492996951 [ 3521.484432] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3521.496050] LustreError: 98458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3522.051677] Lustre: server umount lustre-MDT0001 complete [ 3532.515754] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:40:23 (1786020023) [ 3555.886828] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3570.897468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.251521] LustreError: 103936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3571.345251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3577.582821] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3590.414318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.324709] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3602.502650] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3612.898905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3614.087780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3614.093182] Lustre: Skipped 1 previous similar message [ 3623.509953] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3640.622785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3640.859863] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3641.079438] Lustre: lustre-OST0001: new disk, initializing [ 3641.095584] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3641.117252] Lustre: 106257:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3649.958215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3649.979139] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3650.264847] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3655.638635] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3668.540508] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3678.746094] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3685.628762] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 3685.894815] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.913351] Lustre: Skipped 1 previous similar message [ 3691.940813] Lustre: server umount lustre-MDT0001 complete [ 3695.821807] LustreError: 105451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3695.867355] LustreError: 105451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3709.625264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.093612] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3710.138768] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3710.301193] Lustre: 103924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3711.505543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3711.514753] Lustre: Skipped 1 previous similar message [ 3712.143559] LustreError: 107793:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3712.174367] LustreError: 107793:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3712.202701] LustreError: 103928:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3712.412950] LustreError: 107804:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.106@tcp: cfg command failed: rc = -17 [ 3712.436464] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3712.436464] [ 3716.594507] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3718.914324] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3733.598456] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:43:44 (1786020224) [ 3758.077366] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3789.649924] Lustre: 109880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3825.187754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3829.950120] Lustre: 111019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3941.347590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3941.355917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.371766] Lustre: Skipped 2 previous similar messages [ 3941.379263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3941.382126] Lustre: Skipped 7 previous similar messages [ 3944.199470] Lustre: server umount lustre-MDT0000 complete [ 3946.982979] LustreError: 103931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.002786] LustreError: 103931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3948.830938] LustreError: 103918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786020442 with bad export cookie 1607514483492997945 [ 3948.833978] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.854860] LustreError: 103918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3949.230680] Lustre: server umount lustre-MDT0001 complete [ 3964.969955] Lustre: server umount lustre-OST0000 complete [ 3967.457281] Lustre: 93484:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786020445/real 1786020445] req@ffff95bb12d5d880 x1872777362955136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786020461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3967.521047] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.544184] Lustre: Skipped 3 previous similar messages [ 3968.291780] Lustre: 93486:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786020446/real 1786020446] req@ffff95bb3b4d1880 x1872777362955392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786020462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3970.805857] Lustre: server umount lustre-OST0001 complete [ 3979.589452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3993.564386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4006.180722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4020.319949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4054.180539] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4066.953815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.989739] Lustre: lustre-MDT0000: reset Object Index mappings [ 4067.881245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4073.919211] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4085.381521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4085.430984] Lustre: lustre-MDT0001: reset Object Index mappings [ 4093.409023] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4098.006619] Lustre: 116159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4108.337458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4110.771250] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 4112.914477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 4119.883505] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4134.819448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4141.060625] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 4141.063281] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 4145.628633] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4159.976695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4167.433270] Lustre: 118040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4247.766755] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:52:18 (1786020738) [ 4274.145786] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.165980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4274.178226] Lustre: Skipped 2 previous similar messages [ 4274.198343] Lustre: Skipped 6 previous similar messages [ 4277.140829] Lustre: server umount lustre-MDT0000 complete [ 4279.291306] LustreError: 115012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.328515] LustreError: 115012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 4283.038457] LustreError: 114992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786020776 with bad export cookie 1607514483493044831 [ 4283.042957] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4283.065023] LustreError: 114992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4290.224289] Lustre: server umount lustre-MDT0001 complete [ 4301.708882] Lustre: server umount lustre-OST0000 complete [ 4313.611364] Lustre: server umount lustre-OST0001 complete [ 4323.513672] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4339.006841] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4390.002176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.611662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4390.700231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4390.921671] Lustre: lustre-MDT0000: new disk, initializing [ 4391.123364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4391.136952] Lustre: Skipped 3 previous similar messages [ 4391.165317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4396.502806] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4405.413380] Lustre: server umount lustre-MDT0000 complete [ 4417.463550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4417.949191] Lustre: Found index 0 for lustre-MDT0000, updating log [ 4417.971142] Lustre: 122339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4417.991909] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 4424.134056] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4430.506380] Lustre: server umount lustre-MDT0000 complete [ 4444.151732] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:55:34 (1786020934) [ 4446.927670] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 4450.715670] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:55:40 (1786020940) [ 4486.956416] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4501.556443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.927751] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4520.742747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.932724] Lustre: 124414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4521.008957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4521.014395] Lustre: Skipped 1 previous similar message [ 4521.145219] Lustre: lustre-MDT0001: new disk, initializing [ 4521.375623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4521.393747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4529.217482] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4535.599200] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4545.272163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4545.762620] Lustre: lustre-OST0000: new disk, initializing [ 4545.771762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4545.791788] Lustre: 125994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4547.762670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4547.766418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4548.008635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4555.840627] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4578.763343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4579.119484] Lustre: lustre-OST0001: new disk, initializing [ 4579.126779] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4579.139955] Lustre: 126864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4580.457603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4580.538918] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4580.867056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4591.489578] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4605.004576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4610.627169] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5207.187704] Lustre: 133326:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 5207.202692] Lustre: 133326:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 5207.205087] Lustre: 133326:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 5207.215783] Lustre: 133326:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 5207.221951] Lustre: 133326:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 5207.231357] Lustre: 133326:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 5601.310372] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:14:51 (1786022091) [ 5622.242245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5622.253140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.279808] Lustre: Skipped 3 previous similar messages [ 5622.292215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5622.305182] Lustre: Skipped 4 previous similar messages [ 5625.229034] Lustre: server umount lustre-MDT0000 complete [ 5628.405355] LustreError: 124423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5628.435036] LustreError: 124423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5631.054763] LustreError: 124407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786022124 with bad export cookie 1607514483493118716 [ 5631.078569] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5631.082974] LustreError: 124407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5633.508250] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.520930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5633.546872] Lustre: Skipped 4 previous similar messages [ 5633.559789] Lustre: Skipped 4 previous similar messages [ 5638.246186] Lustre: server umount lustre-MDT0001 complete [ 5650.450112] Lustre: server umount lustre-OST0000 complete [ 5669.344897] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5670.209200] Lustre: server umount lustre-OST0001 complete [ 5681.521669] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 5697.568103] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5750.705665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.062141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5751.112966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5751.227986] Lustre: lustre-MDT0000: new disk, initializing [ 5751.324494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5751.332486] Lustre: Skipped 5 previous similar messages [ 5751.353199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5757.226767] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5774.400747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.743222] Lustre: 136425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5774.894947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5774.909100] Lustre: Skipped 1 previous similar message [ 5775.304844] Lustre: lustre-MDT0001: new disk, initializing [ 5775.635374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5775.658735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5782.882531] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5796.407215] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5803.925834] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5811.925802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5812.399784] Lustre: lustre-OST0000: new disk, initializing [ 5812.407513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5812.419476] Lustre: 138396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5813.872567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5813.904522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5814.076244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5820.749415] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5832.672113] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5845.120723] Lustre: lustre-MDD0000: changelog on [ 5848.268775] Lustre: lustre-MDD0001: changelog on [ 8487.425240] Lustre: lustre-MDD0001: changelog off [ 8489.772123] Lustre: 137249:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 8490.600087] Lustre: lustre-MDD0000: changelog off [ 8503.780381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8503.782393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.794598] LustreError: Skipped 1 previous similar message [ 8503.795388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8503.795392] Lustre: Skipped 2 previous similar messages [ 8503.842655] Lustre: Skipped 1 previous similar message [ 8506.367483] Lustre: server umount lustre-OST0000 complete [ 8514.530744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.551155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8514.560745] Lustre: Skipped 4 previous similar messages [ 8517.276255] Lustre: server umount lustre-MDT0000 complete [ 8519.650216] LustreError: 136438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8519.684735] LustreError: 136438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8521.868735] LustreError: 136419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786025015 with bad export cookie 1607514483493946655 [ 8521.870294] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8521.883901] LustreError: 136419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8522.669828] Lustre: server umount lustre-MDT0001 complete [ 8537.605172] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8541.503167] Key type lgssc unregistered [ 8542.010110] LNet: 147651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8542.020109] LNetError: 147651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8542.048268] LNet: Removed LNI 192.168.203.106@tcp [ 8543.599254] Key type .llcrypt unregistered [ 8543.602812] Key type ._llcrypt unregistered [ 8572.633096] Key type ._llcrypt registered [ 8572.646747] Key type .llcrypt registered [ 8572.786648] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 8593.718164] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8595.037796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8595.209523] alg: No test for adler32 (adler32-zlib) [ 8596.577099] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 8597.005062] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8598.952485] Key type lgssc registered [ 8601.144449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8649.630619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8649.675987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8651.134640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8651.192336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8651.307115] Lustre: lustre-MDT0000: new disk, initializing [ 8651.435310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8651.474800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8658.444722] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8678.276909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.616334] Lustre: 150865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8678.829629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8678.836834] Lustre: Skipped 1 previous similar message [ 8679.097378] Lustre: lustre-MDT0001: new disk, initializing [ 8679.345821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8679.617844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8679.656481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8687.522156] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8702.617367] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8711.533516] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8721.806578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8722.384051] Lustre: lustre-OST0000: new disk, initializing [ 8722.391667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8722.412378] Lustre: 152834:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8722.570953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.745146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8729.752303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8729.895853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8732.262076] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8743.621074] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8752.288057] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_module obdecho/obdecho [ 8755.089885] LustreError: 153840:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 8755.106164] LustreError: 153840:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 8758.435876] LustreError: 153936:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 8776.163627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.169181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8776.191802] Lustre: Skipped 1 previous similar message [ 8776.203611] Lustre: Skipped 1 previous similar message [ 8782.017534] Lustre: server umount lustre-MDT0000 complete [ 8786.403750] LustreError: 150876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.440531] LustreError: 150876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8787.000434] LustreError: 154280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786025280 with bad export cookie 9784864548410650920 [ 8787.016130] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8787.027884] LustreError: 154280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8787.943162] Lustre: server umount lustre-MDT0001 complete [ 8803.791235] Lustre: server umount lustre-OST0000 complete [ 8837.682848] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 8854.842638] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8913.907121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8914.222446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8914.241411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8914.352756] Lustre: lustre-MDT0000: new disk, initializing [ 8914.468462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8914.488030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8919.937646] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8933.380119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.641545] Lustre: 157678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8933.813781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8933.821079] Lustre: Skipped 1 previous similar message [ 8933.980904] Lustre: lustre-MDT0001: new disk, initializing [ 8934.177378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8934.241122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8934.253482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8939.203655] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8953.048714] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8962.440026] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8976.657414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8978.135163] Lustre: lustre-OST0000: new disk, initializing [ 8978.144203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8978.157524] Lustre: 159650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8978.287062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8983.774103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8983.799629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8984.012611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8992.924979] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9007.081271] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9015.669480] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9016.231603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9024.189411] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9024.333591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9030.117580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9030.125372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.148670] LustreError: Skipped 1 previous similar message [ 9030.183444] Lustre: Skipped 1 previous similar message [ 9030.192353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9030.223071] Lustre: Skipped 1 previous similar message [ 9035.248148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9035.271081] Lustre: Skipped 1 previous similar message [ 9040.353584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9041.376112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9042.147050] Lustre: server umount lustre-OST0000 complete [ 9050.595120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.630284] Lustre: Skipped 1 previous similar message [ 9050.650372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.664886] Lustre: Skipped 2 previous similar messages [ 9060.839263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9060.851813] Lustre: Skipped 3 previous similar messages [ 9062.368252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9062.912032] Lustre: server umount lustre-MDT0000 complete [ 9065.954818] LustreError: 157690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9065.977979] LustreError: 157690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9068.503054] LustreError: 157669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786025562 with bad export cookie 9784864548410651879 [ 9068.513743] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9068.518984] LustreError: 157669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9069.560853] Lustre: server umount lustre-MDT0001 complete [ 9074.726844] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:12:45 (1786025565) [ 9077.345677] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 9079.822285] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:12:51 (1786025571) [ 9092.138853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9092.776805] LustreError: 162074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.891373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9098.220835] LustreError: 162075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9100.208907] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9102.307973] LustreError: 162074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9111.821584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9117.748669] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9127.060923] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9136.858175] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9147.612683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9148.131509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9148.145591] Lustre: Skipped 1 previous similar message [ 9158.254544] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9169.660958] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9180.919187] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 9182.463526] Lustre: server umount lustre-MDT0000 complete [ 9183.201791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.208201] LustreError: 164077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9183.222016] Lustre: Skipped 2 previous similar messages [ 9183.268905] LustreError: 164077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9188.431298] LustreError: 162056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786025682 with bad export cookie 9784864548410652761 [ 9188.449693] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9188.450060] LustreError: 162056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9193.449590] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.472160] Lustre: Skipped 1 previous similar message [ 9195.564462] Lustre: server umount lustre-MDT0001 complete [ 9205.009207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9223.611628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9224.098374] LustreError: 165925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9224.137346] LustreError: 165925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9224.160263] LustreError: 165920:0:(llog.c:1655:llog_backup()) MGC192.168.203.106@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 9224.182294] Lustre: 165920:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.106@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 9257.677349] LustreError: 165926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9257.721421] LustreError: 165926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 9265.145238] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9265.148178] Lustre: Skipped 2 previous similar messages [ 9285.360390] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 9285.364272] Lustre: 165920:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 9322.215878] LustreError: 165925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9322.243715] LustreError: 165925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 40 previous similar messages [ 9326.103678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9331.213037] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9334.644987] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9347.114852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9354.735260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 9366.524151] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9366.643923] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9377.000629] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9379.215712] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9387.569637] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9390.631953] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9422.819045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9422.842893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9422.855943] Lustre: Skipped 2 previous similar messages [ 9422.870158] Lustre: Skipped 2 previous similar messages [ 9424.876472] Lustre: server umount lustre-MDT0000 complete [ 9429.750494] LustreError: 162054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786025923 with bad export cookie 9784864548410659215 [ 9429.756028] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9429.775162] LustreError: 162054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9430.794343] Lustre: server umount lustre-MDT0001 complete [ 9446.286641] Lustre: server umount lustre-OST0000 complete [ 9473.848251] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 9487.765291] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 9543.814971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.254949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9544.343618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9544.514576] Lustre: lustre-MDT0000: new disk, initializing [ 9544.671723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.695535] Lustre: Skipped 1 previous similar message [ 9544.721725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9551.629766] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9565.609067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.916064] Lustre: 171036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9566.503210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9574.145917] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9590.872839] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9606.766650] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9617.555937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9617.946282] Lustre: lustre-OST0000: new disk, initializing [ 9617.951653] Lustre: Skipped 1 previous similar message [ 9617.959834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9617.966514] Lustre: Skipped 2 previous similar messages [ 9617.973301] Lustre: 173008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9624.191039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9624.211377] Lustre: Skipped 1 previous similar message [ 9624.218822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9624.389826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9627.959986] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9642.481858] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9651.755681] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9652.179930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9662.417494] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9663.093349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9670.631545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9670.637582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9670.638558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9670.653361] Lustre: Skipped 1 previous similar message [ 9675.572977] Lustre: server umount lustre-OST0000 complete [ 9686.741415] Lustre: server umount lustre-MDT0000 complete [ 9691.103196] LustreError: 171029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026184 with bad export cookie 9784864548410676043 [ 9691.108234] LustreError: 171048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9691.132512] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9691.135387] LustreError: 171029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9691.187177] LustreError: 171048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 9691.507150] Lustre: server umount lustre-MDT0001 complete [ 9709.502382] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:23:18 (1786026198) [ 9728.037193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.364570] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9753.281733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9763.296448] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9775.282879] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9782.752695] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9793.471382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9804.938827] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9821.627135] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9845.081035] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:25:34 (1786026334) [ 9855.458104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9855.461356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9855.505395] Lustre: Skipped 3 previous similar messages [ 9859.191481] Lustre: server umount lustre-OST0000 complete [ 9871.163594] Lustre: server umount lustre-MDT0000 complete [ 9877.304298] LustreError: 175408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026371 with bad export cookie 9784864548410676925 [ 9877.341425] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9878.558065] Lustre: server umount lustre-MDT0001 complete [ 9901.171887] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 9904.763664] Key type lgssc unregistered [ 9905.176824] LNet: 179187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9905.190088] LNetError: 179187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9905.220539] LNet: Removed LNI 192.168.203.106@tcp [ 9906.853078] Key type .llcrypt unregistered [ 9906.859065] Key type ._llcrypt unregistered [ 9931.825624] Key type ._llcrypt registered [ 9931.829910] Key type .llcrypt registered [ 9931.973465] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 9933.563197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9933.637556] alg: No test for adler32 (adler32-zlib) [ 9934.705460] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 9935.004947] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 9936.704844] Key type lgssc registered [ 9938.440643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9948.174125] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9948.259151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9964.000496] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9969.122423] LustreError: 180431:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 9974.305160] LustreError: 180431:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 9974.318786] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 9974.326409] LustreError: MGC192.168.203.106@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 9974.338866] LustreError: 180431:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 9974.352076] LustreError: 180431:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 9974.360657] LustreError: 180431:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9974.463751] Lustre: server umount lustre-OST0000 complete [ 9974.470901] LustreError: 180431:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9985.447460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9987.265733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9993.636864] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9997.801465] LustreError: 180933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9997.816573] LustreError: 180933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10002.923426] LustreError: 180932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10004.543165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10005.125567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10014.961408] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10030.185796] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10038.479827] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10047.095244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10047.388902] LustreError: 182752:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [10047.404705] Lustre: 182752:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.106@tcp: invalid local config log lustre-OST0000: rc = -22 [10047.597195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10055.082330] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10066.495852] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10071.521660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10071.531762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10071.547527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10073.572345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10073.581160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10073.611444] Lustre: Skipped 2 previous similar messages [10076.199551] Lustre: server umount lustre-MDT0000 complete [10078.693049] LustreError: 180933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10078.721520] LustreError: 180933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10081.106094] LustreError: 180913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026574 with bad export cookie 17341840471185151621 [10081.123091] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10081.133997] LustreError: 180913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10081.804453] Lustre: server umount lustre-MDT0001 complete [10097.466810] Lustre: server umount lustre-OST0000 complete [10109.246684] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [10120.547806] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10174.697114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10175.109096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10175.189423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10175.369735] Lustre: lustre-MDT0000: new disk, initializing [10175.584781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10175.621461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10182.119740] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10197.625106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10197.822126] Lustre: 186280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10197.893961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10197.900918] Lustre: Skipped 1 previous similar message [10197.959361] Lustre: lustre-MDT0001: new disk, initializing [10198.057195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10198.102894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10198.119319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10203.828883] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10215.938232] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10223.025386] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10231.152097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10231.596727] Lustre: lustre-OST0000: new disk, initializing [10231.616991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10231.626059] Lustre: 188250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10231.743667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10240.544969] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10241.070170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10241.095789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10241.239138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10251.336093] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10257.033528] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10257.272505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10262.868576] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10263.094858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10267.105119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10267.107345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10267.113905] LustreError: Skipped 1 previous similar message [10267.118813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10267.157952] Lustre: Skipped 2 previous similar messages [10270.177718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10270.189587] Lustre: Skipped 1 previous similar message [10271.600741] Lustre: server umount lustre-OST0000 complete [10279.909803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10279.918704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10279.932211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10279.943195] Lustre: Skipped 1 previous similar message [10282.844151] Lustre: server umount lustre-MDT0000 complete [10285.537710] LustreError: 186291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.558928] LustreError: 186291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10288.562849] LustreError: 186272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026782 with bad export cookie 17341840471185152342 [10288.568263] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10288.583747] LustreError: 186272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10289.311072] Lustre: server umount lustre-MDT0001 complete [10301.939232] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:33:13 (1786026793) [10330.264105] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [10335.041292] Key type lgssc unregistered [10335.435454] LNet: 191218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10335.447423] LNetError: 191218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10336.485772] LNet: Removed LNI 192.168.203.106@tcp [10337.835102] Key type .llcrypt unregistered [10337.839711] Key type ._llcrypt unregistered [10365.203695] Key type ._llcrypt registered [10365.210776] Key type .llcrypt registered [10365.409127] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10367.666979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10367.863966] alg: No test for adler32 (adler32-zlib) [10369.327485] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [10369.780544] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [10371.844251] Key type lgssc registered [10374.362271] Lustre: Echo OBD driver; http://www.lustre.org/ [10383.467268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10383.493277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.938850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10391.908918] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10395.129567] LustreError: 192440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.158663] LustreError: 192440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10400.229410] LustreError: 192441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.345967] LustreError: 192440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.403090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.066601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10411.659967] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10422.781855] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10430.486786] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10440.607260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10440.832407] Lustre: *** cfs_fail_loc=131e, val=0*** [10440.843342] LustreError: 194307:0:(llog.c:1696:llog_backup()) MGC192.168.203.106@tcp: failed to backup log lustre-OST0000: rc = -28 [10440.850456] Lustre: 194307:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.106@tcp: can't backup local config lustre-OST0000: rc = -28 [10440.974727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10448.704040] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10459.810994] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10464.745272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10464.772157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10464.782931] Lustre: Skipped 2 previous similar messages [10469.869036] LustreError: 193356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10470.509459] Lustre: server umount lustre-MDT0000 complete [10474.977913] LustreError: 192441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10475.015082] LustreError: 192441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10477.403331] LustreError: 192420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026971 with bad export cookie 16135686935593749400 [10477.415282] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10477.426406] LustreError: 192420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10478.222271] Lustre: server umount lustre-MDT0001 complete [10495.145700] Lustre: server umount lustre-OST0000 complete [10510.328909] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [10529.570896] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10583.034196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10583.440411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10583.478211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10583.619984] Lustre: lustre-MDT0000: new disk, initializing [10583.721432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10583.750970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10589.938200] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10605.789119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10606.026191] Lustre: 197857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10606.116825] Lustre: *** cfs_fail_loc=131e, val=0*** [10606.121902] Lustre: Skipped 1 previous similar message [10606.135682] LustreError: 198639:0:(llog.c:1696:llog_backup()) MGC192.168.203.106@tcp: failed to backup log lustre-MDT0001: rc = -28 [10606.148313] LustreError: 198639:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [10606.161617] Lustre: 198639:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.106@tcp: failed to copy new config lustre-MDT0001: rc = -28 [10606.208384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10606.216225] Lustre: Skipped 1 previous similar message [10606.379244] Lustre: lustre-MDT0001: new disk, initializing [10606.576810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10606.625705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10606.640747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10615.110945] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10638.776036] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10648.232881] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10659.229269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10659.592813] Lustre: *** cfs_fail_loc=131e, val=0*** [10659.594842] Lustre: Skipped 1 previous similar message [10659.596948] LustreError: 199804:0:(llog.c:1696:llog_backup()) MGC192.168.203.106@tcp: failed to backup log lustre-OST0000: rc = -28 [10659.601357] LustreError: 199804:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [10659.604729] Lustre: 199804:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.106@tcp: failed to copy new config lustre-OST0000: rc = -28 [10659.611768] Lustre: 199804:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [10659.631567] Lustre: lustre-OST0000: new disk, initializing [10659.638732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10659.648933] Lustre: 199829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10659.757812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10669.246743] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10669.634913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10669.659590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10669.957346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10682.691785] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10689.577300] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10689.952956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10696.878671] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10697.240986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10700.769764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10700.770069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10700.779661] LustreError: Skipped 1 previous similar message [10700.796694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10700.796702] Lustre: Skipped 2 previous similar messages [10705.907162] LustreError: 199814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.044572] Lustre: server umount lustre-OST0000 complete [10711.023730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10711.035337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10711.051419] Lustre: Skipped 2 previous similar messages [10711.072203] Lustre: Skipped 2 previous similar messages [10716.133204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10716.142471] Lustre: Skipped 1 previous similar message [10721.253381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10721.266892] Lustre: Skipped 1 previous similar message [10725.345184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10725.755713] Lustre: server umount lustre-MDT0000 complete [10726.369391] LustreError: 197868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10726.396493] LustreError: 197868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10731.076404] LustreError: 197847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786027224 with bad export cookie 16135686935593750121 [10731.079426] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10731.091368] LustreError: 197847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10732.115561] Lustre: server umount lustre-MDT0001 complete [10744.067055] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:40:35 (1786027235) [10770.526753] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10790.656329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10790.950653] LustreError: 202986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.967975] LustreError: 202986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10790.999150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10796.332672] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10808.944512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10816.732919] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10820.618962] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10830.129029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10830.743421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10830.748689] Lustre: Skipped 1 previous similar message [10839.734832] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10853.185182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10853.522547] Lustre: lustre-OST0001: new disk, initializing [10853.538630] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10853.549760] Lustre: 205295:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10862.788922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10862.806507] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10862.972477] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10864.380478] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10878.484385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10887.764242] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10890.763293] Lustre: DEBUG MARKER: ADD OST3 [10897.342823] loop: module loaded [10897.417091] loop0: detected capacity change from 0 to 204800000 [10897.768256] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10897.781625] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10897.803120] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10905.670923] Lustre: DEBUG MARKER: STOP OST3 [10906.277630] loop0: detected capacity change from 0 to 204800000 [10913.137961] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10913.473682] Lustre: lustre-OST0003: new disk, initializing [10913.483326] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [10913.497813] Lustre: 207354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10913.721868] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [10913.735030] Lustre: Skipped 1 previous similar message [10915.166952] Lustre: *** cfs_fail_loc=2109, val=2*** [10915.419461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [10915.434377] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [10915.483956] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [10917.216891] LustreError: 207467:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [10917.256259] Lustre: Failing over lustre-OST0003 [10917.684563] Lustre: server umount lustre-OST0003 complete [10920.457605] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10920.461878] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.491184] LustreError: 205764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10920.512093] LustreError: 205764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10923.143957] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10935.290133] Lustre: DEBUG MARKER: START OST3 again [10943.883207] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10944.179624] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [10945.325487] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [10945.971630] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10945.986305] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [10951.476651] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10977.089890] Lustre: Failing over lustre-OST0003 [10977.291247] Lustre: server umount lustre-OST0003 complete [10980.339306] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10980.360938] Lustre: Skipped 2 previous similar messages [10998.758503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10998.761648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10998.786209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11004.651741] Lustre: server umount lustre-MDT0000 complete [11009.585627] LustreError: 206064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786027503 with bad export cookie 16135686935593751003 [11009.591552] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11009.601611] LustreError: 206064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11016.639631] Lustre: server umount lustre-MDT0001 complete [11021.989930] Lustre: server umount lustre-OST0000 complete [11026.488606] Lustre: server umount lustre-OST0001 complete [11035.407381] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [11046.103585] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11094.562084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11094.978651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11095.039776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11095.298393] Lustre: lustre-MDT0000: new disk, initializing [11095.551840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11095.565263] Lustre: Skipped 1 previous similar message [11095.627737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11102.159331] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11116.762717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11116.964343] Lustre: 212768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11117.032700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11117.041670] Lustre: Skipped 1 previous similar message [11117.172410] Lustre: lustre-MDT0001: new disk, initializing [11117.519380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11117.537729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11123.136928] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11133.499238] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11141.271654] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11150.659550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11151.299354] Lustre: lustre-OST0000: new disk, initializing [11151.313399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11151.347426] Lustre: 214737:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11152.528990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11152.553613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11152.721625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11160.848520] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11176.501810] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11184.422155] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11184.714158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11190.695729] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11190.995744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11193.826131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11193.828725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11193.834826] LustreError: Skipped 1 previous similar message [11193.838102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11193.838107] Lustre: Skipped 8 previous similar messages [11193.869302] Lustre: Skipped 6 previous similar messages [11199.594436] Lustre: server umount lustre-OST0000 complete [11204.577637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11211.075730] Lustre: server umount lustre-MDT0000 complete [11214.312244] LustreError: 212779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11214.329697] LustreError: 212779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 32 previous similar messages [11215.941230] LustreError: 212759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786027709 with bad export cookie 16135686935593753292 [11215.949974] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11215.961068] LustreError: 212759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11216.838785] Lustre: server umount lustre-MDT0001 complete [11222.032252] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:48:33 (1786027713) [11245.082549] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [11258.082813] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11310.340282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.772598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11310.844797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11310.976889] Lustre: lustre-MDT0000: new disk, initializing [11311.081240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11317.287668] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11330.221281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11330.387889] Lustre: 219352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11330.845143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11336.030028] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11347.874032] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11357.095937] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11367.029106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11367.441551] Lustre: 221323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11367.615307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11367.617714] Lustre: Skipped 4 previous similar messages [11369.365737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11369.537492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11376.062736] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11389.036674] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11397.496790] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11397.899391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11404.781545] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11405.264093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11410.401155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11410.404128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.417869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11410.426641] Lustre: Skipped 3 previous similar messages [11410.451813] Lustre: Skipped 7 previous similar messages [11415.668952] Lustre: server umount lustre-OST0000 complete [11428.661702] Lustre: server umount lustre-MDT0000 complete [11433.949469] LustreError: 221322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786027927 with bad export cookie 16135686935593754174 [11433.955347] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11433.970529] LustreError: 221322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11435.173365] Lustre: server umount lustre-MDT0001 complete [11449.098400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11458.685714] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11470.592987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11477.984347] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11488.883186] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11496.693648] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11504.626941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11513.089355] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11531.870882] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11546.083188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11546.090438] LustreError: Skipped 2 previous similar messages [11546.098394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11546.115668] Lustre: Skipped 2 previous similar messages [11546.118936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11546.126787] Lustre: Skipped 3 previous similar messages [11551.267746] Lustre: server umount lustre-OST0000 complete [11571.168249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11571.632719] Lustre: server umount lustre-MDT0000 complete [11575.793427] LustreError: 223544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.815127] LustreError: 223544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [11576.006944] LustreError: 223524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786028069 with bad export cookie 16135686935593755056 [11576.016062] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11576.019668] LustreError: 223524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11576.693547] Lustre: server umount lustre-MDT0001 complete [11673.314314] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11677.744985] Key type lgssc unregistered [11678.240358] LNet: 227209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11678.250984] LNetError: 227209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11678.280118] LNet: Removed LNI 192.168.203.106@tcp [11679.673128] Key type .llcrypt unregistered [11679.674567] Key type ._llcrypt unregistered [11720.429482] Key type ._llcrypt registered [11720.430830] Key type .llcrypt registered [11720.638574] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [11744.399485] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11746.632575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11746.662203] alg: No test for adler32 (adler32-zlib) [11748.117772] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [11748.599953] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [11750.362480] Key type lgssc registered [11752.748651] Lustre: Echo OBD driver; http://www.lustre.org/ [11806.780919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11806.808717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11808.130909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11808.245220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11808.401869] Lustre: lustre-MDT0000: new disk, initializing [11808.542761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11808.580728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11814.890704] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11828.622359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11828.837534] Lustre: 230576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11828.961571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11828.968739] Lustre: Skipped 1 previous similar message [11829.111670] Lustre: lustre-MDT0001: new disk, initializing [11829.289288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11829.353314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11829.374265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11835.390518] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11848.978570] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11858.467411] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11867.545457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11868.019878] Lustre: lustre-OST0000: new disk, initializing [11868.026976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11868.036814] Lustre: 232543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11868.186148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11876.252423] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11876.426123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11876.455468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11876.629477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11890.232360] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11897.656687] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11898.249242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11906.300241] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11906.727584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11912.675465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11912.684193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11912.694474] LustreError: Skipped 1 previous similar message [11912.698876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11912.718633] Lustre: Skipped 1 previous similar message [11912.735762] Lustre: Skipped 1 previous similar message [11916.873868] Lustre: server umount lustre-OST0000 complete [11922.914135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11922.939896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11922.947440] Lustre: Skipped 1 previous similar message [11928.046327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11929.432974] Lustre: server umount lustre-MDT0000 complete [11935.588805] LustreError: 230566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786028429 with bad export cookie 17108918524296310570 [11935.591666] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11935.603979] LustreError: 230566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11936.592586] Lustre: server umount lustre-MDT0001 complete [11965.084015] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11979.870220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.579514] LustreError: 235509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.618663] LustreError: 235509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11980.765612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11985.889244] LustreError: 235510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.596521] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11991.010750] LustreError: 235509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11996.142170] LustreError: 235510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11998.113508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11998.613365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12004.625521] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12007.954404] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12017.802940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12018.241816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12024.991299] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12037.843325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12038.268423] Lustre: lustre-OST0001: new disk, initializing [12038.282404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12038.294260] Lustre: 237817:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12038.481393] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12046.891946] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12046.980355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12046.989851] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12047.209498] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12060.682963] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12070.992989] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12091.121130] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:03:02 (1786028582) [12095.971145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12095.985930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12096.017320] Lustre: Skipped 1 previous similar message [12096.026124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12096.029252] Lustre: Skipped 1 previous similar message [12098.543876] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12098.575612] Lustre: Skipped 2 previous similar messages [12101.187426] Lustre: server umount lustre-MDT0000 complete [12103.650888] LustreError: 238970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12103.687784] LustreError: 238970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12106.956262] LustreError: 235490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786028600 with bad export cookie 17108918524296311452 [12106.967246] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12106.975042] LustreError: 235490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12108.188027] Lustre: server umount lustre-MDT0001 complete [12122.403481] Lustre: server umount lustre-OST0000 complete [12124.969147] Lustre: 228793:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786028602/real 1786028602] req@ffff95ba026d2a00 x1872786350012416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786028618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12125.010157] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12128.162139] Lustre: 228795:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786028605/real 1786028605] req@ffff95bb4190ad80 x1872786350012672/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786028621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12128.925794] Lustre: server umount lustre-OST0001 complete [12139.763461] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [12156.005591] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12207.193076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12207.519544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12207.568538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12207.668349] Lustre: lustre-MDT0000: new disk, initializing [12207.811212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12207.840671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12214.973766] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12227.978826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12228.187978] Lustre: 241807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12228.294770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12228.308554] Lustre: Skipped 1 previous similar message [12228.443844] Lustre: lustre-MDT0001: new disk, initializing [12228.622826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12228.635768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12234.067710] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12246.926654] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12254.604199] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12263.621359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12263.967888] Lustre: lustre-OST0000: new disk, initializing [12263.981201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12263.992645] Lustre: 243779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12265.670985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12265.677854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12265.904397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12272.372665] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12285.525394] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12292.283612] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12292.561569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12299.387703] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12299.704596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12306.915412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12306.916108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12306.928189] LustreError: Skipped 1 previous similar message [12306.931664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12306.931671] Lustre: Skipped 4 previous similar messages [12306.957448] Lustre: Skipped 1 previous similar message [12309.104188] Lustre: server umount lustre-OST0000 complete [12315.617809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12315.621422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12315.627810] Lustre: Skipped 1 previous similar message [12321.499482] Lustre: server umount lustre-MDT0000 complete [12322.277469] LustreError: 241820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12322.315558] LustreError: 241820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12327.707241] LustreError: 243778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786028821 with bad export cookie 17108918524296313587 [12327.723137] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12327.731823] LustreError: 243778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12328.745730] Lustre: server umount lustre-MDT0001 complete [12341.401751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12342.217684] LustreError: 245998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12342.254217] LustreError: 245998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12342.322093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12342.328227] Lustre: Skipped 2 previous similar messages [12347.380597] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12360.390407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12365.858708] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12376.010337] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12383.943306] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12392.828362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12402.618976] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12413.146433] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12561.383365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12561.385792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12561.388298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12561.411502] Lustre: Skipped 3 previous similar messages [12561.416210] Lustre: Skipped 2 previous similar messages [12566.451726] Lustre: server umount lustre-OST0000 complete [12576.226769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12578.139746] Lustre: server umount lustre-MDT0000 complete [12581.867207] LustreError: 246727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12581.885046] LustreError: 246727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12583.332290] LustreError: 245979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786029077 with bad export cookie 17108918524296314469 [12583.340172] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12583.347425] LustreError: 245979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12584.215483] Lustre: server umount lustre-MDT0001 complete [12687.065898] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [12691.313620] Key type lgssc unregistered [12691.891560] LNet: 249606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12691.902277] LNetError: 249606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12692.970393] LNet: Removed LNI 192.168.203.106@tcp [12694.123243] Key type .llcrypt unregistered [12694.128566] Key type ._llcrypt unregistered [12733.590405] Key type ._llcrypt registered [12733.594689] Key type .llcrypt registered [12733.757119] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [12756.067965] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12757.307396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12757.475596] alg: No test for adler32 (adler32-zlib) [12758.608944] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [12758.937771] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [12760.664193] Key type lgssc registered [12762.363237] Lustre: Echo OBD driver; http://www.lustre.org/ [12833.193164] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12850.011959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12850.118546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12851.575524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12851.651651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12851.791251] Lustre: lustre-MDT0000: new disk, initializing [12851.952183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12852.025680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12860.655372] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12880.410571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12880.572728] Lustre: 253708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12880.671356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12880.678155] Lustre: Skipped 1 previous similar message [12880.889576] Lustre: lustre-MDT0001: new disk, initializing [12881.127332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12881.192725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12881.208657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12887.238983] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12893.032757] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12901.365983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12901.885812] Lustre: lustre-OST0000: new disk, initializing [12901.897136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12901.920806] Lustre: 255342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12902.105124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12904.504631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12904.520609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12904.677748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12912.070828] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12927.905816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12928.268055] Lustre: lustre-OST0001: new disk, initializing [12928.281038] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12928.293763] Lustre: 256216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12928.504106] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12935.308557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12935.325939] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12935.505768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12938.609819] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12953.249636] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12962.548581] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12981.733124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12981.745781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12981.766913] Lustre: Skipped 2 previous similar messages [12981.800675] Lustre: Skipped 3 previous similar messages [12985.936845] Lustre: server umount lustre-MDT0000 complete [12991.970172] LustreError: 253715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12991.993324] LustreError: 253715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [12992.031871] LustreError: 253698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786029485 with bad export cookie 8947114125473772788 [12992.041521] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12992.046471] LustreError: 253698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12992.751308] Lustre: server umount lustre-MDT0001 complete [13007.561058] Lustre: server umount lustre-OST0000 complete [13012.448207] Lustre: 251190:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786029490/real 1786029490] req@ffff95bb41a1aa00 x1872787409034368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786029506 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13012.515868] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13012.562322] Lustre: Skipped 1 previous similar message [13013.779332] Lustre: server umount lustre-OST0001 complete [13032.877055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13048.935148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13061.552898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [13079.767083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [13121.491987] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [13135.745533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13135.791705] Lustre: lustre-MDT0000: reset Object Index mappings [13136.544182] LustreError: 261256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13136.673984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13141.985154] LustreError: 261257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13143.567312] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13147.108616] LustreError: 261256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13152.229707] LustreError: 261257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13155.113331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13155.148973] Lustre: lustre-MDT0001: reset Object Index mappings [13155.709152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13162.336227] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13166.733670] Lustre: 262399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [13175.366270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13175.844537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13180.965263] LustreError: 262754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13182.024123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [13184.009423] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13194.565529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13199.880291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [13202.849975] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13213.577498] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13219.373377] Lustre: 264271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [13230.562743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13230.628583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13230.650888] Lustre: Skipped 3 previous similar messages [13232.393849] Lustre: server umount lustre-MDT0000 complete [13235.693937] LustreError: 262370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13235.743851] LustreError: 262370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [13239.144705] LustreError: 261238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786029732 with bad export cookie 8947114125473779578 [13239.147563] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13239.154282] LustreError: 261238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13239.816410] Lustre: server umount lustre-MDT0001 complete [13256.307641] Lustre: server umount lustre-OST0000 complete [13257.184302] Lustre: 251190:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786029734/real 1786029734] req@ffff95bb3c406300 x1872787409092096/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786029750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13257.230551] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13257.251439] Lustre: Skipped 3 previous similar messages [13260.960135] Lustre: 251189:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786029738/real 1786029738] req@ffff95ba12e4a300 x1872787409092352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786029754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13262.317771] Lustre: server umount lustre-OST0001 complete [13289.714418] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:23:00 (1786029780) [13311.319815] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [13328.241660] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [13379.657777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13379.940339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13379.974633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13380.083359] Lustre: lustre-MDT0000: new disk, initializing [13380.179673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13380.185890] Lustre: Skipped 1 previous similar message [13380.216416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13386.999164] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13404.119924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13404.404839] Lustre: 268550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13404.588156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13404.603252] Lustre: Skipped 1 previous similar message [13404.893379] Lustre: lustre-MDT0001: new disk, initializing [13405.150614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13405.179879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13411.627941] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13426.334730] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13434.516577] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13442.906862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13443.461132] Lustre: lustre-OST0000: new disk, initializing [13443.473250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13443.493377] Lustre: 270521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13445.349194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13445.364701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13445.529318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13451.328497] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13466.337365] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13473.599768] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13473.858676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13480.150697] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13480.420404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13486.572301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13486.572587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13486.574098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13486.599483] LustreError: Skipped 1 previous similar message [13486.664160] Lustre: Skipped 1 previous similar message [13491.683637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13491.693723] Lustre: Skipped 2 previous similar messages [13496.803702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13496.818084] Lustre: Skipped 1 previous similar message [13497.313029] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13497.550984] Lustre: server umount lustre-OST0000 complete [13502.435539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13502.447140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13507.041214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13507.043786] Lustre: Skipped 1 previous similar message [13508.988166] Lustre: server umount lustre-MDT0000 complete [13512.161931] LustreError: 268562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13512.185790] LustreError: 268562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13513.554976] LustreError: 268544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786030007 with bad export cookie 8947114125473781664 [13513.558811] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13513.579486] LustreError: 268544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13514.822426] Lustre: server umount lustre-MDT0001 complete [13547.903455] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [13562.171839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13563.074160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13563.085222] Lustre: Skipped 2 previous similar messages [13569.808544] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13577.700889] LustreError: 273482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13577.740068] LustreError: 273482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13582.239686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13589.476900] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13595.307213] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13606.846104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13616.240864] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13627.178743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13627.384740] Lustre: lustre-OST0001: new disk, initializing [13627.389569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13627.399765] Lustre: 275792:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13629.167813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13629.192777] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13629.561874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13636.746892] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13651.946638] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13656.895191] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [13663.123238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [13663.382717] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [13674.977268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13674.995476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13675.031559] Lustre: Skipped 1 previous similar message [13675.042255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13679.777469] Lustre: server umount lustre-MDT0000 complete [13686.271363] LustreError: 273463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786030180 with bad export cookie 8947114125473782553 [13686.288027] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13686.296033] LustreError: 273463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13687.432269] Lustre: server umount lustre-MDT0001 complete [13704.968067] Lustre: server umount lustre-OST0000 complete [13706.720837] Lustre: 251190:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786030184/real 1786030184] req@ffff95bb077f6d80 x1872787409213184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786030200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13706.755356] Lustre: 251190:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13711.004088] Lustre: server umount lustre-OST0001 complete [13740.833957] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [13761.973822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13762.447855] LustreError: 278853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13762.459379] LustreError: 278853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [13762.509372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13762.513963] Lustre: Skipped 3 previous similar messages [13770.884719] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13783.409643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13790.071369] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13793.706332] Lustre: 279994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [13802.112241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13807.868898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [13811.696139] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13822.558340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13828.112200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [13831.451894] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13841.955595] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13846.456872] Lustre: 281872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [13851.531359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [13851.853616] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [13864.109575] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:32:35 (1786030355) [13881.316951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13881.337690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13881.369132] Lustre: Skipped 4 previous similar messages [13881.397355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13881.414396] Lustre: Skipped 4 previous similar messages [13886.355890] Lustre: server umount lustre-MDT0000 complete [13890.722795] LustreError: 281180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786030384 with bad export cookie 8947114125473788895 [13890.734574] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13890.738711] LustreError: 281180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13905.376324] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13905.898200] Lustre: server umount lustre-MDT0001 complete [13917.946364] Lustre: server umount lustre-OST0000 complete [13924.421684] Lustre: server umount lustre-OST0001 complete [13933.936269] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [13949.961423] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14022.161227] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14036.459896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14036.781357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14036.833569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14036.952534] Lustre: lustre-MDT0000: new disk, initializing [14037.038044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14037.040866] Lustre: Skipped 3 previous similar messages [14037.059534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14042.933805] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14057.557526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14057.860315] Lustre: 286165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14058.433684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14065.457501] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14073.458299] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14083.404127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14083.912304] Lustre: 287799:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14085.109123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14092.516485] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14105.788788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14106.055725] Lustre: 288670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14108.044165] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14108.060139] Lustre: Skipped 1 previous similar message [14108.214100] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14113.992430] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14125.918266] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14155.044929] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:37:26 (1786030646) [14159.331383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14159.335391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14159.348765] Lustre: Skipped 8 previous similar messages [14159.364488] Lustre: Skipped 13 previous similar messages [14164.473597] LustreError: 287847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14164.500028] LustreError: 287847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [14164.999247] Lustre: server umount lustre-MDT0000 complete [14169.696816] LustreError: 287798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786030663 with bad export cookie 8947114125473796749 [14169.713356] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14169.718409] LustreError: 287798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14170.912315] Lustre: server umount lustre-MDT0001 complete [14186.532114] Lustre: server umount lustre-OST0000 complete [14191.059490] Lustre: 251190:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786030668/real 1786030668] req@ffff95ba12089180 x1872787409383936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786030684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [14191.123591] Lustre: 251190:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14192.310657] Lustre: server umount lustre-OST0001 complete [14205.498683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14212.020905] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14221.895648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14227.458155] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14241.689630] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14253.588587] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14264.668374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14274.223330] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14284.507085] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14304.288342] loop1: detected capacity change from 0 to 204800000 [14304.584949] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14304.612755] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14304.638573] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14310.989478] loop1: detected capacity change from 0 to 204800000 [14319.320618] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14319.472516] LustreError: 290909:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs21428: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [14319.499845] LustreError: 290909:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs21428 [14319.516812] LustreError: 290909:0:(mgs_handler.c:517:mgs_target_reg()) fs21428-MDT0001: Error checking target index: rc = -13 [14319.540167] LustreError: fs21428-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [14319.551746] LustreError: 294639:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [14319.562899] LustreError: 294639:0:(tgt_mount.c:2129:server_put_super()) no obd fs21428-MDT0001 [14319.570776] LustreError: 294639:0:(tgt_mount.c:116:server_deregister_mount()) fs21428-MDT0001 not registered [14319.640364] Lustre: server umount fs21428-MDT0001 complete [14319.643894] LustreError: 294639:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [14330.512310] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14330.882495] Lustre: Setting parameter fs21428-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs21428-MDT0001 [14330.896735] Lustre: Skipped 1 previous similar message [14330.977121] Lustre: srv-fs21428-MDT0001: No data found on store. Initialize space. [14330.985636] Lustre: Skipped 4 previous similar messages [14331.082989] Lustre: fs21428-MDT0001: new disk, initializing [14331.085968] Lustre: Skipped 3 previous similar messages [14331.214602] Lustre: srv-fs21428-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14331.230420] Lustre: 295108:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21428-MDT0001-mdtlov: can't create new llog: rc = -115 [14331.238215] LustreError: 295108:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21428-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [14336.827991] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14345.598782] Lustre: server umount fs21428-MDT0001 complete [14348.278676] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [14361.059264] loop1: detected capacity change from 0 to 204800000 [14368.229934] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14368.704151] Lustre: 296676:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21428-MDT0001-mdtlov: can't create new llog: rc = -115 [14368.713963] LustreError: 296676:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21428-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [14373.682936] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14380.384285] Lustre: server umount fs21428-MDT0001 complete [14393.087117] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [14394.348677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14394.364661] Lustre: Skipped 3 previous similar messages [14398.878268] Lustre: server umount lustre-MDT0000 complete [14411.890662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14412.025143] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14417.922181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [14419.701700] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14430.937015] loop1: detected capacity change from 0 to 204800000 [14439.888363] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14441.159055] Lustre: 299127:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21428-MDT0001-mdtlov: can't create new llog: rc = -115 [14441.177643] LustreError: 299127:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21428-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [14449.522823] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14455.014779] Lustre: server umount fs21428-MDT0001 complete [14490.867150] Lustre: server umount lustre-OST0000 complete [14499.847376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14499.859727] LustreError: Skipped 1 previous similar message [14504.674519] Lustre: server umount lustre-MDT0000 complete [14510.210735] LustreError: 297169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786031004 with bad export cookie 8947114125473799430 [14510.225649] LustreError: 297169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14510.226444] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14510.915206] Lustre: server umount lustre-MDT0001 complete [14527.556865] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [14531.964302] Key type lgssc unregistered [14532.507648] LNet: 301475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14532.513634] LNetError: 301475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14532.546573] LNet: Removed LNI 192.168.203.106@tcp [14533.691122] Key type .llcrypt unregistered [14533.696162] Key type ._llcrypt unregistered [14559.642459] Key type ._llcrypt registered [14559.648311] Key type .llcrypt registered [14559.826539] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14561.719749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14561.801411] alg: No test for adler32 (adler32-zlib) [14563.111688] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [14563.513200] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [14565.203518] Key type lgssc registered [14566.911915] Lustre: Echo OBD driver; http://www.lustre.org/ [14577.546937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14577.557731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14579.183214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14586.213729] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14588.395314] LustreError: 302751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14588.436366] LustreError: 302751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14593.524593] LustreError: 302752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14598.637338] LustreError: 302751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14599.866038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14600.517018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14606.682251] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14616.504623] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14623.776673] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14632.371397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14632.862926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14640.115657] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14646.189397] LustreError: 304587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14650.491311] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14653.417745] LustreError: 304586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14653.449559] LustreError: 304586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14663.660584] LustreError: 304587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14663.682847] LustreError: 304587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14675.110640] loop1: detected capacity change from 0 to 204800000 [14675.307098] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14675.326198] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14675.355916] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14681.856116] loop1: detected capacity change from 0 to 204800000 [14684.155479] LustreError: 304587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14684.184488] LustreError: 304587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [14689.705990] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14690.023693] Lustre: fs7473-OST0000: new disk, initializing [14690.029669] Lustre: srv-fs7473-OST0000: No data found on store. Initialize space. [14690.043131] Lustre: 306494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14690.156796] Lustre: fs7473-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14698.042168] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14707.243972] Lustre: server umount fs7473-OST0000 complete [14719.970744] LustreError: 304587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14720.003918] LustreError: 304587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [14723.927342] loop1: detected capacity change from 0 to 204800000 [14725.113618] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [14733.656041] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14734.324659] Lustre: fs7473-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14746.491782] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14757.138470] loop2: detected capacity change from 0 to 204800000 [14757.396846] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14757.411873] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14757.434407] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14763.894411] loop2: detected capacity change from 0 to 204800000 [14770.592025] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14770.701365] LustreError: fs7473-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [14770.715836] LustreError: fs7473-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [14770.725724] LustreError: 309539:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [14770.731447] LustreError: 309539:0:(tgt_mount.c:2129:server_put_super()) no obd fs7473-OST0001 [14770.735599] LustreError: 309539:0:(tgt_mount.c:116:server_deregister_mount()) fs7473-OST0001 not registered [14770.762680] Lustre: server umount fs7473-OST0001 complete [14770.765879] LustreError: 309539:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [14784.609357] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14785.035593] Lustre: fs7473-OST0001: new disk, initializing [14785.041782] Lustre: srv-fs7473-OST0001: No data found on store. Initialize space. [14785.058346] Lustre: 310091:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14785.255612] Lustre: fs7473-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14786.533791] LustreError: 304586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14786.546803] LustreError: 304586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [14793.537510] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14804.722652] Lustre: server umount fs7473-OST0000 complete [14815.743895] Lustre: server umount fs7473-OST0001 complete [14830.129672] loop1: detected capacity change from 0 to 204800000 [14838.130979] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14838.687610] Lustre: fs7473-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14848.647179] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14859.790039] loop2: detected capacity change from 0 to 204800000 [14868.474100] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14878.562992] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14878.722579] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [14878.734914] LustreError: Skipped 1 previous similar message [14895.901996] Lustre: server umount fs7473-OST0000 complete [14912.219204] Lustre: server umount fs7473-OST0001 complete [14914.587425] LustreError: 304587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14914.631317] LustreError: 304587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 74 previous similar messages [14932.963093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14932.988877] LustreError: Skipped 1 previous similar message [14933.004053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14933.024625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14935.013914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14935.014037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14935.046266] Lustre: Skipped 1 previous similar message [14938.496634] Lustre: server umount lustre-OST0000 complete [14943.714972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14943.718227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14943.731066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14949.063400] Lustre: server umount lustre-MDT0000 complete [14953.401237] LustreError: 302733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786031447 with bad export cookie 17628840910021369006 [14953.418848] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14953.424196] LustreError: 302733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14953.833982] Lustre: server umount lustre-MDT0001 complete [14968.326344] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [14972.801848] Key type lgssc unregistered [14973.374190] LNet: 316021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14973.381282] LNetError: 316021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14973.436831] LNet: Removed LNI 192.168.203.106@tcp [14974.897219] Key type .llcrypt unregistered [14974.898867] Key type ._llcrypt unregistered [15016.696680] Key type ._llcrypt registered [15016.702331] Key type .llcrypt registered [15016.862643] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [15040.155106] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [15042.519420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15042.595627] alg: No test for adler32 (adler32-zlib) [15043.796224] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [15044.057840] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [15045.784145] Key type lgssc registered [15048.391662] Lustre: Echo OBD driver; http://www.lustre.org/ [15102.110234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15102.171883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15103.588195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15103.691649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15103.904569] Lustre: lustre-MDT0000: new disk, initializing [15104.062226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15104.112018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15110.848946] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15125.004961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15125.209751] Lustre: 319887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15125.276934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15125.280434] Lustre: Skipped 1 previous similar message [15125.405780] Lustre: lustre-MDT0001: new disk, initializing [15125.568315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15125.632753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15125.655350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15131.834346] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15150.194440] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15160.460978] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15170.561107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15171.014249] Lustre: lustre-OST0000: new disk, initializing [15171.016820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15171.046295] Lustre: 321858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15171.167779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15179.331818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15179.348152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15179.503507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15180.129644] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15192.071970] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15194.321218] Lustre: Mounted lustre-client [15195.391141] Lustre: Unmounted lustre-client [15209.954457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15209.974366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15209.985988] Lustre: Skipped 1 previous similar message [15209.997345] Lustre: Skipped 1 previous similar message [15215.083780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15220.192118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15220.196956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15220.211348] Lustre: Skipped 2 previous similar messages [15220.693183] Lustre: server umount lustre-OST0000 complete [15225.316791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15225.330648] Lustre: Skipped 1 previous similar message [15225.331416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15225.341037] Lustre: Skipped 1 previous similar message [15230.434846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15230.439106] Lustre: Skipped 1 previous similar message [15231.568513] Lustre: server umount lustre-MDT0000 complete [15235.956662] LustreError: 319878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786031729 with bad export cookie 17587977933184333002 [15235.978890] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15236.756127] Lustre: server umount lustre-MDT0001 complete [15251.873988] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [15252.064594] LNet: 1 local NIs in recovery (showing 1): 192.168.203.106@tcp [15256.013402] Key type lgssc unregistered [15256.458719] LNet: 324024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15256.468366] LNetError: 324024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15325.101739] LNet: Removed LNI 192.168.203.106@tcp [15326.439259] Key type .llcrypt unregistered [15326.443679] Key type ._llcrypt unregistered [15353.986281] Key type ._llcrypt registered [15353.993219] Key type .llcrypt registered [15354.194928] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [15377.288845] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [15379.200341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15379.323985] alg: No test for adler32 (adler32-zlib) [15380.347312] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [15380.521142] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [15382.144148] Key type lgssc registered [15383.741499] Lustre: Echo OBD driver; http://www.lustre.org/ [15428.938724] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:58:40 (1786031920) [15437.105841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15437.126062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15438.471058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15438.526105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15438.669428] Lustre: lustre-MDT0000: new disk, initializing [15438.792128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15438.836368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15444.461424] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15459.048423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15459.243459] Lustre: 327248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15459.357959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15459.369261] Lustre: Skipped 1 previous similar message [15459.506260] Lustre: lustre-MDT0001: new disk, initializing [15459.690577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15459.745591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15459.758512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15465.294414] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15478.249767] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15485.497294] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15494.385024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15494.829871] Lustre: lustre-OST0000: new disk, initializing [15494.838304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15494.847545] Lustre: 329221:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15494.989959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15502.837033] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15504.459532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15504.475207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15504.563534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15514.337672] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15535.598934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15535.603380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15535.612931] LustreError: Skipped 1 previous similar message [15535.616433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15535.654676] Lustre: Skipped 1 previous similar message [15539.249177] Lustre: server umount lustre-OST0000 complete [15545.827832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15545.848068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15545.851319] Lustre: Skipped 1 previous similar message [15550.946451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15550.959558] Lustre: Skipped 1 previous similar message [15551.289075] Lustre: server umount lustre-MDT0000 complete [15556.821374] LustreError: 329220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786032050 with bad export cookie 4518049949054208870 [15556.832390] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15556.842545] LustreError: 329220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15557.597138] Lustre: server umount lustre-MDT0001 complete [15576.444164] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [15580.680732] Key type lgssc unregistered [15581.084457] LNet: 331164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15581.088385] LNetError: 331164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15582.123502] LNet: Removed LNI 192.168.203.106@tcp [15583.567414] Key type .llcrypt unregistered [15583.572980] Key type ._llcrypt unregistered [15609.278366] Key type ._llcrypt registered [15609.282694] Key type .llcrypt registered [15609.390573] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [15610.467493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15610.494452] alg: No test for adler32 (adler32-zlib) [15611.483858] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [15611.800836] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [15613.520155] Key type lgssc registered [15615.394851] Lustre: Echo OBD driver; http://www.lustre.org/ [15623.736057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15623.766772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15625.215423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15631.305620] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15634.414793] LustreError: 332439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15634.459752] LustreError: 332439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15639.532860] LustreError: 332440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15644.646088] LustreError: 332439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15644.774821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15645.425877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15651.444772] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15659.868129] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15668.107880] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15677.283740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15677.842217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15688.383087] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15703.430771] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15715.942944] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15718.216538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15725.026771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15725.038507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15725.049050] Lustre: Skipped 2 previous similar messages [15725.068064] Lustre: Skipped 2 previous similar messages [15725.819766] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [15730.149866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15730.163635] Lustre: Skipped 2 previous similar messages [15735.265383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15735.276403] Lustre: Skipped 3 previous similar messages [15737.312171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15737.983500] Lustre: server umount lustre-MDT0000 complete [15740.388261] LustreError: 332440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15740.413699] LustreError: 332440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15742.902559] LustreError: 332420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786032236 with bad export cookie 6923276640298470774 [15742.926140] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15745.515060] LustreError: 332436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15745.519837] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15745.541808] LustreError: 332436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15745.570220] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [15745.572829] Lustre: Skipped 1 previous similar message [15756.493381] LustreError: 332434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15756.510994] LustreError: 332434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15756.511694] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [15756.533532] Lustre: Skipped 4 previous similar messages [15757.280338] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15757.877213] Lustre: server umount lustre-MDT0001 complete [15764.308651] Lustre: server umount lustre-OST0000 complete [15779.311962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15786.265330] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15789.760326] Lustre: 336628:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.106@tcp [15792.448759] Lustre: server umount MGS complete [15802.112555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15802.666385] LustreError: 337202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15802.689722] LustreError: 337202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15802.796803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15809.172807] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15818.836973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15819.452882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15825.544593] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15837.036897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15837.677840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15846.250196] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [15854.958220] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15856.579551] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [15864.277896] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15866.196393] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15872.998774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [15873.001966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15873.018461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15887.328159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15887.855420] Lustre: server umount lustre-OST0000 complete [15893.987475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15894.007112] Lustre: Skipped 2 previous similar messages [15899.129170] LustreError: 337203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15899.167291] LustreError: 337203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [15899.410210] Lustre: server umount lustre-MDT0000 complete [15903.642626] LustreError: 337182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786032397 with bad export cookie 6923276640298471614 [15903.654917] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15903.658987] LustreError: 337182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [15904.202879] Lustre: server umount lustre-MDT0001 complete [15919.455337] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [15923.064330] Key type lgssc unregistered [15923.333446] LNet: 340645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15923.336395] LNetError: 340645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15923.357501] LNet: Removed LNI 192.168.203.106@tcp [15924.481899] Key type .llcrypt unregistered [15924.485738] Key type ._llcrypt unregistered [15951.320978] Key type ._llcrypt registered [15951.325514] Key type .llcrypt registered [15951.600806] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [15973.881994] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [15976.014584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15976.122105] alg: No test for adler32 (adler32-zlib) [15977.494809] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [15978.013275] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [15979.816316] Key type lgssc registered [15981.633680] Lustre: Echo OBD driver; http://www.lustre.org/ [16029.824128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16029.865618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16031.412833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16031.463386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16031.688428] Lustre: lustre-MDT0000: new disk, initializing [16031.911446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16032.007545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16039.105821] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [16060.564660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16060.758768] Lustre: 343811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16060.948351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16060.962614] Lustre: Skipped 1 previous similar message [16061.186366] Lustre: lustre-MDT0001: new disk, initializing [16061.408861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16061.493238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16061.524350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16068.864359] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [16082.531849] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16089.840231] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16098.405353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16098.849341] Lustre: lustre-OST0000: new disk, initializing [16098.860183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16098.876623] Lustre: 345782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16099.047271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16104.538757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16104.560119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16104.743136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16106.448241] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [16118.874517] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16124.871064] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16125.204919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16130.504107] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16130.793573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16135.649602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16135.652038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16135.664958] LustreError: Skipped 1 previous similar message [16135.683949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16135.691661] Lustre: Skipped 1 previous similar message [16139.456100] Lustre: server umount lustre-OST0000 complete [16145.889338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16145.907592] Lustre: Skipped 1 previous similar message [16145.930062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16145.943975] Lustre: Skipped 1 previous similar message [16151.010462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16151.025312] Lustre: Skipped 1 previous similar message [16156.131477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16156.134317] Lustre: Skipped 1 previous similar message [16159.201026] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16159.552549] Lustre: server umount lustre-MDT0000 complete [16164.236871] LustreError: 343802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786032658 with bad export cookie 9712306052742324009 [16164.239779] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16164.251280] LustreError: 343802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16164.828431] Lustre: server umount lustre-MDT0001 complete [16176.652954] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:11:07 (1786032667) [16203.392653] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [16217.584367] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [16263.437648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16263.796375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16263.856963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16263.985964] Lustre: lustre-MDT0000: new disk, initializing [16264.130488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16264.158352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16269.908951] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [16281.427481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16281.598208] Lustre: 350547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16281.769645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16281.780415] Lustre: Skipped 1 previous similar message [16281.884402] Lustre: lustre-MDT0001: new disk, initializing [16282.032954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16282.091252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16282.114290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16286.743453] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [16298.998643] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16306.494818] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16316.942650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16317.432267] Lustre: lustre-OST0000: new disk, initializing [16317.438552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16317.451254] Lustre: 352521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16317.600281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16325.823905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16325.840863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16326.078477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16327.882687] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [16340.344468] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16392.680308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16392.683528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16392.696298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16392.704592] Lustre: Skipped 2 previous similar messages [16392.735405] LustreError: Skipped 1 previous similar message [16396.425460] Lustre: server umount lustre-OST0000 complete [16402.920317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16402.925150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16402.939117] Lustre: Skipped 1 previous similar message [16402.962173] Lustre: Skipped 2 previous similar messages [16407.563403] Lustre: server umount lustre-MDT0000 complete [16408.033790] LustreError: 350559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16408.058797] LustreError: 350559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [16411.966191] LustreError: 350539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786032905 with bad export cookie 9712306052742324891 [16411.977381] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16411.983557] LustreError: 350539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16412.505622] Lustre: server umount lustre-MDT0001 complete [16425.992727] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [16429.696670] Key type lgssc unregistered [16430.059593] LNet: 354696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16430.070730] LNetError: 354696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16430.096202] LNet: Removed LNI 192.168.203.106@tcp [16431.158508] Key type .llcrypt unregistered [16431.167105] Key type ._llcrypt unregistered [16566.331265] Key type ._llcrypt registered [16566.333693] Key type .llcrypt registered [16566.550850] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [16567.008337] Key type .llcrypt unregistered [16567.018278] Key type ._llcrypt unregistered [16693.984734] Key type ._llcrypt registered [16693.992168] Key type .llcrypt registered [16694.155522] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [16694.722375] Key type .llcrypt unregistered [16694.726324] Key type ._llcrypt unregistered [16747.205674] Key type ._llcrypt registered [16747.213468] Key type .llcrypt registered [16747.554923] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [16748.239466] Key type .llcrypt unregistered [16748.241475] Key type ._llcrypt unregistered [16788.153782] Key type ._llcrypt registered [16788.156074] Key type .llcrypt registered [16788.282216] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [16788.631444] Key type .llcrypt unregistered [16788.633946] Key type ._llcrypt unregistered [16883.575139] Key type ._llcrypt registered [16883.585948] Key type .llcrypt registered [16883.750876] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [16884.236255] Key type .llcrypt unregistered [16884.238288] Key type ._llcrypt unregistered [16926.109711] Key type ._llcrypt registered [16926.120169] Key type .llcrypt registered [16926.487511] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [16926.973484] Key type .llcrypt unregistered [16926.976364] Key type ._llcrypt unregistered [16967.297305] Key type ._llcrypt registered [16967.300208] Key type .llcrypt registered [16967.469633] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [16967.966302] Key type .llcrypt unregistered [16967.974208] Key type ._llcrypt unregistered [17010.805468] Key type ._llcrypt registered [17010.815559] Key type .llcrypt registered [17010.989270] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [17011.374397] Key type .llcrypt unregistered [17011.381418] Key type ._llcrypt unregistered [17053.242491] Key type ._llcrypt registered [17053.246172] Key type .llcrypt registered [17053.407443] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [17053.815514] Key type .llcrypt unregistered [17053.817770] Key type ._llcrypt unregistered [17097.835503] Key type ._llcrypt registered [17097.837804] Key type .llcrypt registered [17097.953200] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [17098.260359] Key type .llcrypt unregistered [17098.262624] Key type ._llcrypt unregistered [17181.060772] Key type ._llcrypt registered [17181.063166] Key type .llcrypt registered [17181.140980] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [17181.380241] Key type .llcrypt unregistered [17181.385803] Key type ._llcrypt unregistered [17230.691489] Key type ._llcrypt registered [17230.693267] Key type .llcrypt registered [17230.778386] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [17231.309458] Key type .llcrypt unregistered [17231.312468] Key type ._llcrypt unregistered [17276.629367] Key type ._llcrypt registered [17276.638623] Key type .llcrypt registered [17276.934418] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [17277.607387] Key type .llcrypt unregistered [17277.613373] Key type ._llcrypt unregistered [17350.942092] Key type ._llcrypt registered [17350.943783] Key type .llcrypt registered [17351.118877] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [17369.538950] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [17370.456864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17370.494275] alg: No test for adler32 (adler32-zlib) [17371.910790] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [17372.249818] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [17373.960133] Key type lgssc registered [17376.005388] Lustre: Echo OBD driver; http://www.lustre.org/ [17424.768789] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17424.863377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17426.340279] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [17426.388838] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [17426.594784] Lustre: tst250fs-MDT0000: new disk, initializing [17426.727648] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17426.758860] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17432.653528] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [17447.351747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17447.522920] Lustre: 375228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17447.657983] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [17447.664766] Lustre: Skipped 1 previous similar message [17447.810817] Lustre: tst250fs-MDT0001: new disk, initializing [17448.077421] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17448.135618] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17448.148695] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17453.981498] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [17460.596765] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [17468.039156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17468.253295] Lustre: tst250fs-OST0000: new disk, initializing [17468.257032] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [17468.261521] Lustre: 376861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17468.348593] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17473.668699] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17473.681918] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17473.860164] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17477.918900] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [17492.124720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17492.499613] Lustre: tst250fs-OST0001: new disk, initializing [17492.508672] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [17492.520360] Lustre: 377733:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17492.720061] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [17498.341185] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [17498.365427] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [17498.471284] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [17500.715173] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [17508.744171] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [17511.346158] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [17511.356275] Lustre: Skipped 1 previous similar message [17534.436509] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17534.441253] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [17534.473375] Lustre: Skipped 1 previous similar message [17534.495657] Lustre: Skipped 3 previous similar messages [17539.553173] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [17539.574114] Lustre: Skipped 3 previous similar messages [17541.107496] Lustre: server umount tst250fs-MDT0000 complete [17547.044253] LustreError: 375220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786034040 with bad export cookie 8056405692834230656 [17547.060961] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17547.061850] LustreError: 375220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17547.578911] Lustre: server umount tst250fs-MDT0001 complete [17562.851495] Lustre: server umount tst250fs-OST0000 complete [17565.683701] Lustre: 373450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786034043/real 1786034043] req@ffff95bb4154e680 x1872792247148928/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786034059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17565.743790] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17565.779356] Lustre: Skipped 2 previous similar messages [17568.992203] Lustre: 373451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786034046/real 1786034046] req@ffff95bb37922d80 x1872792247149184/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786034062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17570.274609] Lustre: 373448:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786034048/real 1786034048] req@ffff95bb05b5e680 x1872792247149312/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786034064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17571.414260] Lustre: server umount tst250fs-OST0001 complete [17603.868995] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [17618.058600] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [17671.170794] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:36:02 (1786034162) [17673.002666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [17705.294725] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [17723.416660] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [17765.806745] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 17457 sec ========= 12:37:37 (1786034257) [17768.502069] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:37:39 (1786034259) === [17772.936826] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:37:44 (1786034264) === [17835.584585] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [17839.591297] Key type lgssc unregistered [17840.076492] LNet: 386898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17840.086766] LNetError: 386898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17840.108409] LNet: Removed LNI 192.168.203.106@tcp [17841.269122] Key type .llcrypt unregistered [17841.274118] Key type ._llcrypt unregistered