[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490096450 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003116] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007977] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010149] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013081] *** VALIDATE selinux *** [ 0.021792] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026681] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028002] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030110] *** VALIDATE tmpfs *** [ 0.032288] *** VALIDATE proc *** [ 0.033226] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039013] Speculative Store Bypass: Vulnerable [ 0.042340] debug: unmapping init [mem 0xffffffff88459000-0xffffffff88460fff] [ 0.044208] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045767] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053318] rcu: Hierarchical SRCU implementation. [ 0.055478] smp: Bringing up secondary CPUs ... [ 0.056644] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060428] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150637] node 0 deferred pages initialised in 86ms [ 0.154112] devtmpfs: initialized [ 0.155250] x86/mm: Memory block size: 128MB [ 0.158953] gcov: version magic: 0x41383552 [ 0.162083] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165392] pinctrl core: initialized pinctrl subsystem [ 0.167218] [ 0.167698] ************************************************************* [ 0.169014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171014] ** ** [ 0.174015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175009] ** ** [ 0.177014] ** This means that this kernel is built to expose internal ** [ 0.180015] ** IOMMU data structures, which may compromise security on ** [ 0.182011] ** your system. ** [ 0.185014] ** ** [ 0.187010] ** If you see this message and you are not debugging the ** [ 0.189011] ** kernel, report this immediately to your vendor! ** [ 0.190011] ** ** [ 0.192012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194023] ************************************************************* [ 0.196686] NET: Registered protocol family 16 [ 0.199404] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204116] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208128] cpuidle: using governor menu [ 0.209824] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212436] PCI: Using configuration type 1 for base access [ 0.214136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.222133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225144] cryptd: max_cpu_qlen set to 1000 [ 0.226225] ACPI: Added _OSI(Module Device) [ 0.228029] ACPI: Added _OSI(Processor Device) [ 0.230015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235980] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.244047] ACPI: Interpreter enabled [ 0.245060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247014] ACPI: Using IOAPIC for interrupt routing [ 0.248144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252437] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.262356] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.264035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.275396] acpiphp: Slot [2] registered [ 0.277106] acpiphp: Slot [5] registered [ 0.279139] acpiphp: Slot [6] registered [ 0.280221] acpiphp: Slot [7] registered [ 0.282123] acpiphp: Slot [8] registered [ 0.283118] acpiphp: Slot [9] registered [ 0.285182] acpiphp: Slot [10] registered [ 0.287124] acpiphp: Slot [3] registered [ 0.288091] acpiphp: Slot [4] registered [ 0.289090] acpiphp: Slot [11] registered [ 0.290518] acpiphp: Slot [12] registered [ 0.292086] acpiphp: Slot [13] registered [ 0.293141] acpiphp: Slot [14] registered [ 0.295105] acpiphp: Slot [15] registered [ 0.296097] acpiphp: Slot [16] registered [ 0.297099] acpiphp: Slot [17] registered [ 0.299101] acpiphp: Slot [18] registered [ 0.300118] acpiphp: Slot [19] registered [ 0.302088] acpiphp: Slot [20] registered [ 0.303131] acpiphp: Slot [21] registered [ 0.305106] acpiphp: Slot [22] registered [ 0.306081] acpiphp: Slot [23] registered [ 0.307115] acpiphp: Slot [24] registered [ 0.308096] acpiphp: Slot [25] registered [ 0.310073] acpiphp: Slot [26] registered [ 0.311090] acpiphp: Slot [27] registered [ 0.313110] acpiphp: Slot [28] registered [ 0.315161] acpiphp: Slot [29] registered [ 0.317141] acpiphp: Slot [30] registered [ 0.319112] acpiphp: Slot [31] registered [ 0.321055] PCI host bridge to bus 0000:00 [ 0.323025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.326024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340423] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344053] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.353909] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.359014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.365020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.367016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370607] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.372918] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379425] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.403017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.407820] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.413983] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.458650] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.472018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.526293] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.532016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.543016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.562019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.570077] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.579024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.588026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.608026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.622110] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.632018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.641016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.658019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.669754] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.677023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.685024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.705025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.716191] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.718414] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.722439] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.723497] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.726218] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.734049] iommu: Default domain type: Passthrough [ 0.736442] SCSI subsystem initialized [ 0.738170] ACPI: bus type USB registered [ 0.740143] usbcore: registered new interface driver usbfs [ 0.742137] usbcore: registered new interface driver hub [ 0.744130] usbcore: registered new device driver usb [ 0.746192] pps_core: LinuxPPS API ver. 1 registered [ 0.748016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.752085] PTP clock support registered [ 0.754091] EDAC MC: Ver: 3.0.0 [ 0.757058] PCI: Using ACPI for IRQ routing [ 0.759258] NetLabel: Initializing [ 0.760011] NetLabel: domain hash size = 128 [ 0.762016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.764091] NetLabel: unlabeled traffic allowed by default [ 0.766134] vgaarb: loaded [ 0.768303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.770017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.775294] clocksource: Switched to clocksource kvm-clock [ 0.887777] VFS: Disk quotas dquot_6.6.0 [ 0.889552] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.892297] *** VALIDATE ramfs *** [ 0.893604] *** VALIDATE hugetlbfs *** [ 0.895186] pnp: PnP ACPI init [ 0.897610] pnp: PnP ACPI: found 6 devices [ 0.924887] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.928989] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.931326] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.933633] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.936153] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.938602] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.941512] NET: Registered protocol family 2 [ 0.944235] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.949302] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.952945] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.957772] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.962149] TCP: Hash tables configured (established 65536 bind 65536) [ 0.964810] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.968331] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971304] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.974510] NET: Registered protocol family 1 [ 0.977386] RPC: Registered named UNIX socket transport module. [ 0.979333] RPC: Registered udp transport module. [ 0.981101] RPC: Registered tcp transport module. [ 0.982839] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985292] NET: Registered protocol family 44 [ 0.986943] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.989103] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.991110] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.993508] PCI: CLS 0 bytes, default 64 [ 0.995218] Unpacking initramfs... [ 2.458982] debug: unmapping init [mem 0xffff97a53cc54000-0xffff97a53ffbffff] [ 2.469458] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.472082] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.476490] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.989889] Initialise system trusted keyrings [ 2.992221] Key type blacklist registered [ 2.994498] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.006773] zbud: loaded [ 3.010105] *** VALIDATE nfs *** [ 3.011450] *** VALIDATE nfs4 *** [ 3.013211] pstore: using deflate compression [ 3.017149] Platform Keyring initialized [ 3.122914] NET: Registered protocol family 38 [ 3.127292] Key type asymmetric registered [ 3.129183] Asymmetric key parser 'x509' registered [ 3.131142] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.134436] io scheduler mq-deadline registered [ 3.136035] io scheduler kyber registered [ 3.137551] io scheduler bfq registered [ 3.139617] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.142483] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.145052] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.147722] ACPI: Power Button [PWRF] [ 3.154210] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.161854] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.176501] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.184151] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.206210] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.234725] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.263960] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.269921] Non-volatile memory driver v1.3 [ 3.271838] Linux agpgart interface v0.103 [ 3.306836] virtio_blk virtio1: [vda] 147160 512-byte logical blocks (75.3 MB/71.9 MiB) [ 3.310465] vda: detected capacity change from 0 to 75345920 [ 3.325273] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.328687] vdb: detected capacity change from 0 to 1073741824 [ 3.344569] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.347667] vdc: detected capacity change from 0 to 2621440000 [ 3.366510] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.369096] vdd: detected capacity change from 0 to 2621440000 [ 3.389536] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.392871] vde: detected capacity change from 0 to 4294967296 [ 3.414394] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.416970] vdf: detected capacity change from 0 to 4294967296 [ 3.424733] libphy: Fixed MDIO Bus: probed [ 3.430069] usbcore: registered new interface driver usbserial_generic [ 3.432701] usbserial: USB Serial support registered for generic [ 3.435593] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.439567] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.441513] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.443972] mousedev: PS/2 mouse device common for all mice [ 3.446734] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.448592] rtc_cmos 00:05: RTC can wake from S4 [ 3.452934] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.452976] rtc_cmos 00:05: registered as rtc0 [ 3.458834] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.459282] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.464854] intel_pstate: CPU model not supported [ 3.468059] hid: raw HID events driver (C) Jiri Kosina [ 3.470253] usbcore: registered new interface driver usbhid [ 3.472372] usbhid: USB HID core driver [ 3.473949] drop_monitor: Initializing network drop monitor service [ 3.476520] Initializing XFRM netlink socket [ 3.478803] NET: Registered protocol family 10 [ 3.481757] Segment Routing with IPv6 [ 3.483390] NET: Registered protocol family 17 [ 3.485784] mpls_gso: MPLS GSO support [ 3.492379] RAS: Correctable Errors collector initialized. [ 3.494543] AVX version of gcm_enc/dec engaged. [ 3.496335] AES CTR mode by8 optimization enabled [ 3.571390] sched_clock: Marking stable (3571362956, 0)->(4485813587, -914450631) [ 3.575266] registered taskstats version 1 [ 3.577389] Loading compiled-in X.509 certificates [ 3.578900] zswap: loaded using pool lzo/zbud [ 3.603743] Key type big_key registered [ 3.616684] Key type encrypted registered [ 3.618518] ima: No TPM chip found, activating TPM-bypass! [ 3.620821] ima: Allocated hash algorithm: sha1 [ 3.622571] ima: No architecture policies found [ 3.624572] evm: Initialising EVM extended attributes: [ 3.626530] evm: security.selinux [ 3.627759] evm: security.ima [ 3.628769] evm: security.capability [ 3.629909] evm: HMAC attrs: 0x1 [ 3.632315] rtc_cmos 00:05: setting system clock to 2026-09-11 12:17:44 UTC (1789129064) [ 3.638759] debug: unmapping init [mem 0xffffffff89403000-0xffffffff895fffff] [ 3.642180] debug: unmapping init [mem 0xffffffff88182000-0xffffffff88458fff] [ 3.651151] Write protecting the kernel read-only data: 28672k [ 3.654671] debug: unmapping init [mem 0xffffffff86803000-0xffffffff869fffff] [ 3.657987] debug: unmapping init [mem 0xffffffff87114000-0xffffffff871fffff] [ 3.694952] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.703834] systemd[1]: Detected virtualization kvm. [ 3.705698] systemd[1]: Detected architecture x86-64. [ 3.707583] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.738766] systemd[1]: No hostname configured. [ 3.740302] systemd[1]: Set hostname to . [ 3.744001] random: systemd: uninitialized urandom read (16 bytes read) [ 3.746515] systemd[1]: Initializing machine ID from random generator. [ 3.802678] random: ln: uninitialized urandom read (6 bytes read) [ 3.871453] random: systemd: uninitialized urandom read (16 bytes read) [ 3.874192] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.878687] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.883312] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook...[ 4.442427] random: fast init done [ 4.491753] device-mapper: uevent: version 1.0.3 [ 4.494155] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.280719] virtio_net virtio0 ens2: renamed from eth0 [ 5.501776] scsi host0: ata_piix [ 5.514766] scsi host1: ata_piix [ 5.516857] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.519405] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.784609] dracut-initqueue[603]: RTNETLINK answers: File exists [ 10.135563] random: crng init done [ 10.137176] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.570267] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.786454] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.068408] SELinux: Disabled at runtime. [ 12.129993] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.139173] systemd[1]: Detected virtualization kvm. [ 12.141276] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.680556] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.684711] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.690358] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.694712] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.697992] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.707174] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.724557] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 12.819828] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.139513] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.517258] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.609790] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.758330] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.790090] EDAC sbridge: Ver: 1.1.2 [ 15.646517] Key type dns_resolver registered [ 15.952689] NFS: Registering the id_resolver key type [ 15.954950] Key type id_resolver registered [ 15.956576] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg422-server login: [ 57.559796] hrtimer: interrupt took 3929299 ns [ 89.287724] libcfs: loading out-of-tree module taints kernel. [ 89.422717] Key type ._llcrypt registered [ 89.426382] Key type .llcrypt registered [ 89.594884] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 109.951269] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 111.694461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.707274] alg: No test for adler32 (adler32-zlib) [ 113.557992] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 115.042713] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 116.967219] Key type lgssc registered [ 119.477956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 140.185905] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 185.711653] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 201.033759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.104165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.459516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.544338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 202.752310] Lustre: lustre-MDT0000: new disk, initializing [ 202.908677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.964502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 208.150462] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.944952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.117528] Lustre: 6534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.148979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 223.152865] Lustre: Skipped 1 previous similar message [ 223.315415] Lustre: lustre-MDT0001: new disk, initializing [ 223.425565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.462706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 223.484396] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.949257] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.685376] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.308714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.251619] Lustre: lustre-OST0000: new disk, initializing [ 245.259085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.272482] Lustre: 8468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.356820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 247.346471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 247.358493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 247.447710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 251.394259] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.706317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.891318] Lustre: lustre-OST0001: new disk, initializing [ 266.894706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 266.900337] Lustre: 9541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 266.998406] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 273.522465] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.033067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.047576] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.183753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 285.551118] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.348811] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 301.670818] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing check_logdir /tmp/testlogs/ [ 307.739212] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing yml_node [ 311.599791] Lustre: DEBUG MARKER: Client: 2.17.57.133 [ 314.015829] Lustre: DEBUG MARKER: MDS: 2.17.57.133 [ 316.223506] Lustre: DEBUG MARKER: OSS: 2.17.57.133 [ 318.107053] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 08:22:56 EDT 2026 [ 335.526699] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 337.240442] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 356.835423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 356.843501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.873882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 357.346736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.359025] Lustre: Skipped 1 previous similar message [ 359.430869] Lustre: server umount lustre-MDT0000 complete [ 363.166871] LustreError: 6526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129424 with bad export cookie 2852860624131403626 [ 363.168908] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 363.183908] LustreError: 6526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 367.583620] LustreError: 6539:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.584367] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.585580] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 367.585592] Lustre: Skipped 4 previous similar messages [ 367.603493] LustreError: 6539:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 367.662650] Lustre: Skipped 2 previous similar messages [ 368.608332] LustreError: 10661:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.608357] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.628389] LustreError: 10661:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 368.634590] Lustre: Skipped 2 previous similar messages [ 369.660332] Lustre: server umount lustre-MDT0001 complete [ 373.182260] Lustre: server umount lustre-OST0000 complete [ 377.097682] Lustre: server umount lustre-OST0001 complete [ 385.081727] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 398.055618] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 438.392751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.570533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 438.600861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 438.702385] Lustre: lustre-MDT0000: new disk, initializing [ 438.768144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.788115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 443.986935] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 455.836539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.931777] Lustre: 15099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 455.988086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 455.992471] Lustre: Skipped 1 previous similar message [ 456.071378] Lustre: lustre-MDT0001: new disk, initializing [ 456.173877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 456.249923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 456.266177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 461.335472] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 471.725628] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 478.679628] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 485.931654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 486.333378] Lustre: lustre-OST0000: new disk, initializing [ 486.337367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 486.349517] Lustre: 17068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 486.473347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.427986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 488.443559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 488.599308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 493.289728] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 503.322438] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 508.232211] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 508.443891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 513.550059] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 513.855638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 519.137704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 519.142807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.158589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.177853] Lustre: Skipped 1 previous similar message [ 524.259978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 524.264028] Lustre: Skipped 1 previous similar message [ 530.399267] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 530.540535] Lustre: server umount lustre-OST0000 complete [ 534.496676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.507552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 534.508918] Lustre: Skipped 1 previous similar message [ 534.515203] Lustre: Skipped 3 previous similar messages [ 548.319596] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 548.626090] Lustre: server umount lustre-MDT0000 complete [ 549.858753] LustreError: 15108:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.884973] LustreError: 15108:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 553.013728] LustreError: 15089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129613 with bad export cookie 2852860624131405159 [ 553.033292] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 553.439181] Lustre: server umount lustre-MDT0001 complete [ 560.505765] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:26:58 (1789129618) [ 581.356950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.152679] LustreError: 19945:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.272949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.250839] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 587.753156] LustreError: 19946:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.335162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.710906] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 609.114676] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 614.355519] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.640580] Lustre: Failing over lustre-MDT0000 [ 616.854804] Lustre: server umount lustre-MDT0000 complete [ 617.443072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 617.453325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.460771] LustreError: 19945:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.474647] Lustre: Skipped 1 previous similar message [ 617.498279] LustreError: 19945:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 633.823610] Lustre: 3666:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789129678/real 1789129678] req@ffff97a483e34000 x1876037913966336/t0(0) o400->MGC192.168.204.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789129694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 633.838235] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.846446] LustreError: 20673:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.861213] LustreError: 20673:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 635.667203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.075791] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2797659f5f0d3aed [ 644.094856] Lustre: MGC192.168.204.122@tcp: Connection restored to 0@lo (at 0@lo) [ 644.623921] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 649.698831] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 649.715200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 649.781848] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 649.884083] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 659.865848] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 665.058757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.058778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 665.060290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 665.071172] Lustre: Skipped 1 previous similar message [ 665.074655] Lustre: Skipped 5 previous similar messages [ 669.353478] Lustre: server umount lustre-MDT0000 complete [ 671.203018] LustreError: 19945:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.245229] LustreError: 19945:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 673.777834] LustreError: 19925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129734 with bad export cookie 2852860624131406573 [ 673.778402] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.787212] LustreError: 19925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 674.578868] Lustre: server umount lustre-MDT0001 complete [ 684.009916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.795177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 684.812520] Lustre: Skipped 2 previous similar messages [ 690.075546] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 697.817477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.301616] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 711.749835] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 719.431574] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 721.919435] Lustre: Failing over lustre-MDT0000 [ 722.404121] Lustre: server umount lustre-MDT0000 complete [ 723.940471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 723.952333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.302980] LustreError: 23287:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 739.328029] LustreError: 23287:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 740.319170] Lustre: 3665:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789129784/real 1789129784] req@ffff97a483d0b480 x1876037913996288/t0(0) o400->MGC192.168.204.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789129800 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 740.351460] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 744.179646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.584995] LustreError: 3663:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97a483d53480 x1876037913999232/t0(0) o250->MGC192.168.204.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 752.116854] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 757.228438] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 757.246556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 757.271049] Lustre: Skipped 1 previous similar message [ 757.320414] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 758.566985] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 772.070243] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 778.209073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 778.215877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 778.218467] Lustre: Skipped 3 previous similar messages [ 782.256500] Lustre: server umount lustre-MDT0000 complete [ 786.389330] LustreError: 23268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129847 with bad export cookie 2852860624131407511 [ 786.394852] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 786.397378] LustreError: 23268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 786.975733] Lustre: server umount lustre-MDT0001 complete [ 796.234902] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:30:54 (1789129854) [ 817.510515] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 827.409788] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 870.624789] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 882.889253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 883.105516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 883.141617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 883.240312] Lustre: lustre-MDT0000: new disk, initializing [ 883.344516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.351238] Lustre: Skipped 2 previous similar messages [ 883.375118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 888.343941] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 899.599176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.762241] Lustre: 29669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 900.036738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 904.519215] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 913.773697] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 920.171659] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 928.214655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.751934] Lustre: 31639:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 930.362494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 930.441482] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 930.587858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 930.633195] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 937.019523] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 945.795967] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 950.916922] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 971.749521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.751739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 971.776950] Lustre: Skipped 4 previous similar messages [ 971.783071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 971.818399] Lustre: Skipped 3 previous similar messages [ 975.734238] Lustre: server umount lustre-OST0000 complete [ 993.247421] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 993.486042] Lustre: server umount lustre-MDT0000 complete [ 996.330950] LustreError: 29681:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.353796] LustreError: 29681:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 997.653553] LustreError: 29661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130058 with bad export cookie 2852860624131407910 [ 997.660268] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.664183] LustreError: 29661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 998.189342] Lustre: server umount lustre-MDT0001 complete [ 1009.493614] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 1011.784659] Key type lgssc unregistered [ 1011.997914] LNet: 33621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.002515] LNetError: 33621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.021235] LNet: Removed LNI 192.168.204.122@tcp [ 1013.010307] Key type .llcrypt unregistered [ 1013.011956] Key type ._llcrypt unregistered [ 1037.496411] Key type ._llcrypt registered [ 1037.499773] Key type .llcrypt registered [ 1037.659525] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 1039.976931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1040.028780] alg: No test for adler32 (adler32-zlib) [ 1041.106644] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 1041.447822] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 1043.151180] Key type lgssc registered [ 1044.433294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1054.565701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1054.576392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.276600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.186149] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1066.470037] LustreError: 35038:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.482521] LustreError: 35038:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1071.586053] LustreError: 35037:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.111438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.799324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1078.382912] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1086.311682] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1091.859845] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1099.524898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1099.954056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.501420] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1116.087379] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1117.366348] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1130.976744] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:36:29 (1789130189) [ 1132.689700] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1134.522344] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1140.956745] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:36:39 (1789130199) [ 1149.412455] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:36:48 (1789130208) [ 1150.395907] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1158.061497] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1158.066893] Lustre: Skipped 2 previous similar messages [ 1159.899532] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:36:58 (1789130218) [ 1163.145946] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1163.152538] Lustre: Skipped 3 previous similar messages [ 1180.015936] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1180.027497] Lustre: Skipped 6 previous similar messages [ 1183.729529] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:37:21 (1789130241) [ 1186.418370] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1190.367576] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1198.138724] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:37:37 (1789130257) [ 1200.117415] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1200.122945] Lustre: Skipped 3 previous similar messages [ 1213.363748] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:37:52 (1789130272) [ 1230.311723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1230.314557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.320165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.320332] Lustre: Skipped 1 previous similar message [ 1230.328845] Lustre: Skipped 1 previous similar message [ 1232.015838] Lustre: server umount lustre-OST0000 complete [ 1236.451740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1236.468823] LustreError: Skipped 1 previous similar message [ 1236.475124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.499307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1240.544163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1240.546593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.953603] Lustre: server umount lustre-MDT0000 complete [ 1245.507625] LustreError: 35020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130306 with bad export cookie 12915693292221998694 [ 1245.525056] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.937280] Lustre: server umount lustre-MDT0001 complete [ 1257.228731] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 1260.153568] Key type lgssc unregistered [ 1260.492745] LNet: 42890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.506829] LNetError: 42890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1260.538665] LNet: Removed LNI 192.168.204.122@tcp [ 1261.520911] Key type .llcrypt unregistered [ 1261.523190] Key type ._llcrypt unregistered [ 1280.322417] Key type ._llcrypt registered [ 1280.325312] Key type .llcrypt registered [ 1280.466969] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 1296.510844] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 1297.908434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1297.974253] alg: No test for adler32 (adler32-zlib) [ 1299.063712] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 1299.402833] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 1301.065669] Key type lgssc registered [ 1302.406793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1338.980856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1339.031114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1340.416887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1340.471082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1340.580320] Lustre: lustre-MDT0000: new disk, initializing [ 1340.669866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.695231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1346.522601] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1358.700397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.826284] Lustre: 46045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1358.920489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1358.924827] Lustre: Skipped 1 previous similar message [ 1358.998625] Lustre: lustre-MDT0001: new disk, initializing [ 1359.123203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1359.153210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1359.163721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1363.494514] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1373.300823] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1379.985604] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1385.973486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1386.281794] Lustre: lustre-OST0000: new disk, initializing [ 1386.289530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1386.295212] Lustre: 48012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1386.380186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.560551] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1392.217298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1392.224333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1392.347038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1399.111938] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1402.972760] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1403.203412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1407.963249] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1408.313819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1413.091932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.107996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1413.110632] Lustre: Skipped 1 previous similar message [ 1413.116081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.119502] LustreError: Skipped 1 previous similar message [ 1413.139170] Lustre: Skipped 1 previous similar message [ 1416.507221] Lustre: server umount lustre-OST0000 complete [ 1420.776215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1420.797488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.829409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.329728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.329957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.792399] Lustre: server umount lustre-MDT0000 complete [ 1430.213566] LustreError: 46036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130491 with bad export cookie 1591595104224677750 [ 1430.226596] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1430.234239] LustreError: 46036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1430.492971] Lustre: server umount lustre-MDT0001 complete [ 1440.745677] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:41:39 (1789130499) [ 1447.699138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.191885] LustreError: 50473:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.221403] LustreError: 50473:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1448.279524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.061895] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1453.537477] LustreError: 50474:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.664068] LustreError: 50473:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.365862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.727458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1466.018306] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1473.494239] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1479.429413] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1485.678280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1485.895574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.694451] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1498.653993] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1502.721898] LustreError: 53001:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1503.538453] Lustre: Modifying parameter general.timeout=1 in log params [ 1505.560934] Lustre: Modifying parameter general.timeout=3 in log params [ 1505.567831] Lustre: Skipped 1 previous similar message [ 1507.602076] Lustre: Modifying parameter general.timeout=5 in log params [ 1507.606890] Lustre: Skipped 1 previous similar message [ 1511.785894] Lustre: Modifying parameter general.timeout=10 in log params [ 1511.792503] Lustre: Skipped 4 previous similar messages [ 1520.468662] Lustre: Modifying parameter general.timeout=18 in log params [ 1520.471221] Lustre: Skipped 7 previous similar messages [ 1536.742789] Lustre: Modifying parameter general.timeout=36 in log params [ 1536.747954] Lustre: Skipped 17 previous similar messages [ 1568.894909] Lustre: Modifying parameter general.timeout=73 in log params [ 1568.905513] Lustre: Skipped 36 previous similar messages [ 1628.956983] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:44:47 (1789130687) [ 1631.831232] Lustre: 59942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1633.585827] Lustre: 60040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1634.428601] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1634.434410] Lustre: Skipped 60 previous similar messages [ 1637.182912] Lustre: 60253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1637.193719] Lustre: 60253:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1666.466732] Lustre: 61816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1666.476461] Lustre: 61816:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1666.503936] LustreError: 61816:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1670.211484] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:45:28 (1789130728) [ 1685.986631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.988455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1686.006844] Lustre: Skipped 2 previous similar messages [ 1686.010866] Lustre: Skipped 2 previous similar messages [ 1688.660649] Lustre: server umount lustre-MDT0000 complete [ 1691.108260] LustreError: 50474:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.130450] LustreError: 50474:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1692.026609] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130752 with bad export cookie 1591595104224678632 [ 1692.034432] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.037353] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1692.403373] Lustre: server umount lustre-MDT0001 complete [ 1706.780711] Lustre: server umount lustre-OST0000 complete [ 1733.793228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.105917] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1734.433840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.712818] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1746.874405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.951477] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1746.977644] Lustre: 64352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1747.048345] LustreError: 64363:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.064747] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1761.189977] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1766.973859] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1774.434654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1774.700037] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1775.109825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.115617] Lustre: Skipped 1 previous similar message [ 1781.550862] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1790.900474] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1792.749553] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1792.758464] Lustre: Skipped 19 previous similar messages [ 1792.809429] Lustre: 66919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1800.672160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1800.681160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.685133] LustreError: Skipped 1 previous similar message [ 1800.687640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.687649] Lustre: Skipped 1 previous similar message [ 1800.724285] Lustre: Skipped 1 previous similar message [ 1806.730228] Lustre: server umount lustre-OST0000 complete [ 1810.917582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.927428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.933306] Lustre: Skipped 1 previous similar message [ 1810.945056] Lustre: Skipped 3 previous similar messages [ 1825.248746] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1825.520381] Lustre: server umount lustre-MDT0000 complete [ 1826.274808] LustreError: 64363:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.302180] LustreError: 64363:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1829.438153] LustreError: 64345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130890 with bad export cookie 1591595104224681635 [ 1829.442806] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.458587] LustreError: 64345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1829.744334] Lustre: server umount lustre-MDT0001 complete [ 1842.088580] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 1845.074955] Key type lgssc unregistered [ 1845.566490] LNet: 68137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.581822] LNetError: 68137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.638553] LNet: Removed LNI 192.168.204.122@tcp [ 1847.673184] Key type .llcrypt unregistered [ 1847.675419] Key type ._llcrypt unregistered [ 1873.133675] Key type ._llcrypt registered [ 1873.135774] Key type .llcrypt registered [ 1873.291600] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 1874.602465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1874.676090] alg: No test for adler32 (adler32-zlib) [ 1875.779540] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 1875.991719] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 1877.687212] Key type lgssc registered [ 1878.870666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1887.110800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1887.125796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.611399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.546491] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1898.977481] LustreError: 69552:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1899.010353] LustreError: 69552:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1900.056252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.696769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.580577] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1914.982677] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1921.614816] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1928.851131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1929.386966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.662829] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1943.226195] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1951.359986] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1961.445713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.447255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1961.466964] Lustre: Skipped 2 previous similar messages [ 1961.476830] Lustre: Skipped 2 previous similar messages [ 1965.531728] Lustre: server umount lustre-MDT0000 complete [ 1966.565955] LustreError: 69553:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.584788] LustreError: 69553:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1968.742841] LustreError: 69534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131029 with bad export cookie 12333836170368317189 [ 1968.750643] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.761812] LustreError: 69534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1969.157438] Lustre: server umount lustre-MDT0001 complete [ 1983.121680] Lustre: server umount lustre-OST0000 complete [ 2009.333608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.589239] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2009.646197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2010.066314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.154760] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2023.730709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.851297] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2023.895909] Lustre: 74423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2024.430540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2029.146924] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2036.980747] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2042.524845] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2048.879573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2049.092110] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2049.449700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.620605] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2063.791979] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2065.784172] Lustre: 76999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2067.635278] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2074.081698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2074.090418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.100815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2074.593644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2078.695641] Lustre: server umount lustre-OST0000 complete [ 2084.843953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.847620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.855830] Lustre: Skipped 2 previous similar messages [ 2084.883111] Lustre: Skipped 2 previous similar messages [ 2088.791382] Lustre: server umount lustre-MDT0000 complete [ 2089.958918] LustreError: 75556:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.976174] LustreError: 75556:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2093.147412] LustreError: 74413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131154 with bad export cookie 12333836170368318036 [ 2093.148969] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2093.159772] LustreError: 74413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2093.461441] Lustre: server umount lustre-MDT0001 complete [ 2105.174808] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 2108.454316] Key type lgssc unregistered [ 2108.886782] LNet: 78217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.897180] LNetError: 78217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.923167] LNet: Removed LNI 192.168.204.122@tcp [ 2109.814538] Key type .llcrypt unregistered [ 2109.816540] Key type ._llcrypt unregistered [ 2136.845643] Key type ._llcrypt registered [ 2136.849564] Key type .llcrypt registered [ 2136.956552] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 2138.311033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2138.442028] alg: No test for adler32 (adler32-zlib) [ 2139.547959] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 2139.820347] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 2141.535164] Key type lgssc registered [ 2142.766643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2150.718534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2150.732335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.108871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.788652] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2161.633980] LustreError: 79636:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.647157] LustreError: 79636:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2165.134438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2166.188214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2171.662967] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2180.014745] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2186.377242] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2193.632343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2194.381458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.468441] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2207.898608] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2216.600554] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2217.525246] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2219.285442] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2219.289743] Lustre: Skipped 1 previous similar message [ 2221.835401] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2221.837879] Lustre: Skipped 2 previous similar messages [ 2226.168603] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2226.182302] Lustre: Skipped 4 previous similar messages [ 2234.890979] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2234.896509] Lustre: Skipped 8 previous similar messages [ 2251.410376] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2251.415590] Lustre: Skipped 17 previous similar messages [ 2283.976756] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2283.991599] Lustre: Skipped 36 previous similar messages [ 2313.430751] Lustre: 87736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2316.637108] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:56:14 (1789131374) [ 2318.525338] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2320.718473] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:56:19 (1789131379) [ 2328.548789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.550708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2328.569538] Lustre: Skipped 1 previous similar message [ 2328.584702] Lustre: Skipped 1 previous similar message [ 2333.664726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2333.677338] Lustre: Skipped 1 previous similar message [ 2338.798306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2338.809331] Lustre: Skipped 1 previous similar message [ 2339.295692] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2339.676124] Lustre: server umount lustre-OST0000 complete [ 2343.913263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.921035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.928760] Lustre: Skipped 1 previous similar message [ 2343.941613] Lustre: Skipped 1 previous similar message [ 2349.027968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.044344] Lustre: Skipped 1 previous similar message [ 2358.239317] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2358.440463] Lustre: server umount lustre-MDT0000 complete [ 2359.278826] LustreError: 79636:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.315680] LustreError: 79636:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2362.824904] LustreError: 79616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131423 with bad export cookie 12022910598136942713 [ 2362.828606] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.842650] LustreError: 79616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2363.302751] Lustre: server umount lustre-MDT0001 complete [ 2377.990150] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 2381.118190] Key type lgssc unregistered [ 2381.316604] LNet: 88991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2381.324110] LNetError: 88991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2381.355496] LNet: Removed LNI 192.168.204.122@tcp [ 2382.405780] Key type .llcrypt unregistered [ 2382.407501] Key type ._llcrypt unregistered [ 2386.624151] Key type ._llcrypt registered [ 2386.626678] Key type .llcrypt registered [ 2386.754028] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2390.839989] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules [ 2391.527832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2391.561340] alg: No test for adler32 (adler32-zlib) [ 2392.707353] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 2393.008253] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 2394.759217] Key type lgssc registered [ 2395.658347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2396.678833] LustreError: 90194:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2415.949643] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 2419.172144] LustreError: 90194:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2425.268731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2425.291544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.870334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.761727] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2437.091649] LustreError: 91193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.118481] LustreError: 91193:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2441.107794] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:58:19 (1789131499) [ 2442.209265] LustreError: 91194:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.075339] LustreError: 91225:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2446.091492] LustreError: 91225:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2446.099453] LustreError: 91225:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2446.421183] Lustre: server umount lustre-MDT0000 complete [ 2460.839311] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 2464.127361] Key type lgssc unregistered [ 2464.503841] LNet: 92724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.520735] LNetError: 92724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2464.542171] LNet: Removed LNI 192.168.204.122@tcp [ 2465.631754] Key type .llcrypt unregistered [ 2465.636077] Key type ._llcrypt unregistered [ 2484.566582] Key type ._llcrypt registered [ 2484.568952] Key type .llcrypt registered [ 2484.669078] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 2485.829067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2485.899946] alg: No test for adler32 (adler32-zlib) [ 2487.053689] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 2487.327556] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 2489.023185] Key type lgssc registered [ 2490.110151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2498.310380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2498.328091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.038338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.447733] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2509.292367] LustreError: 93945:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.318801] LustreError: 93945:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2513.960153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2514.605323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2518.333407] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2525.252722] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2530.899436] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2537.064899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2537.790145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.668499] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2551.820108] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2596.961455] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:00:55 (1789131655) [ 2598.798968] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2600.655834] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:00:59 (1789131659) [ 2606.563349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2606.579116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.597730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2607.587981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2607.594334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.624609] Lustre: Skipped 1 previous similar message [ 2611.297915] Lustre: server umount lustre-MDT0000 complete [ 2612.713474] LustreError: 93942:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.728950] LustreError: 93942:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2614.738422] LustreError: 93924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131675 with bad export cookie 2994774071464020087 [ 2614.750170] LustreError: 93924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2614.751669] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2615.130900] Lustre: server umount lustre-MDT0001 complete [ 2629.133346] Lustre: server umount lustre-OST0000 complete [ 2641.788395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.509803] LustreError: 98230:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.762422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.184696] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2648.036805] LustreError: 98231:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.157463] LustreError: 98231:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.491523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.268739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2662.011645] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2670.011824] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2675.909281] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2690.276380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.678667] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2690.687967] LustreError: 98217:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2690.756125] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2690.764839] LustreError: 100315:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2690.775582] LustreError: 100315:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2690.786211] LustreError: 100315:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2690.921786] Lustre: server umount lustre-OST0000 complete [ 2690.924819] LustreError: 100315:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2699.138636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2699.321383] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2699.329213] LustreError: 98219:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2699.349716] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2699.359622] LustreError: 100781:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2699.367360] LustreError: 100781:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2699.373146] LustreError: 100781:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2699.439319] Lustre: server umount lustre-OST0000 complete [ 2699.447715] LustreError: 100781:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2708.997619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2709.218850] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2709.231772] Lustre: Found index 0 for lustre-OST0000, updating log [ 2709.250838] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2709.400397] Lustre: lustre-OST0000: new disk, initializing [ 2709.405331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2709.411992] Lustre: 101315:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2709.733291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.580785] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2720.679559] Lustre: Failing over lustre-OST0000 [ 2720.919656] Lustre: server umount lustre-OST0000 complete [ 2726.817037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.822860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.839888] Lustre: Skipped 1 previous similar message [ 2726.849531] Lustre: Skipped 2 previous similar messages [ 2732.001659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2732.010250] Lustre: Skipped 1 previous similar message [ 2737.124602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2737.131676] Lustre: Skipped 1 previous similar message [ 2739.167119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2739.324368] Lustre: server umount lustre-MDT0000 complete [ 2742.244179] LustreError: 98230:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.997601] LustreError: 98212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131803 with bad export cookie 2994774071464022824 [ 2743.004805] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.019866] LustreError: 98212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2743.475906] Lustre: server umount lustre-MDT0001 complete [ 2752.842810] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:03:31 (1789131811) [ 2776.234889] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 2786.627916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.859867] LustreError: 103687:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.869195] LustreError: 103687:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2786.925992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.060388] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2805.925832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2812.281832] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2815.438872] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2822.075781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2823.014473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.019341] Lustre: Skipped 1 previous similar message [ 2823.027236] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2829.108514] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2831.206811] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2831.211988] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:10 [ 2831.227896] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2831.230227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2831.244282] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2831.259221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2837.377939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2837.456940] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2837.542596] Lustre: lustre-OST0001: new disk, initializing [ 2837.546420] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2837.552263] Lustre: 106006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2838.473932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2838.486827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2838.558392] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2843.997570] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2850.964448] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2856.558402] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2861.846746] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.22@tcp (stopping) [ 2863.076267] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2863.087765] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.359163] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2875.608735] Lustre: server umount lustre-MDT0001 complete [ 2877.154933] LustreError: 103683:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.179614] LustreError: 103683:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2884.769184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2884.860519] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2884.865378] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2884.920528] Lustre: 103676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2886.462595] LustreError: 107554:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2886.481220] LustreError: 107554:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2886.492431] LustreError: 103679:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2886.609471] LustreError: 107568:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.122@tcp: cfg command failed: rc = -17 [ 2886.620486] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2886.620486] [ 2890.493356] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2890.745153] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2899.763485] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:05:58 (1789131958) [ 2919.023843] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 2938.519945] Lustre: 109618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2961.950853] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2964.808993] Lustre: 110752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3054.048345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3054.059116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.068675] Lustre: Skipped 2 previous similar messages [ 3054.074270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3054.078607] Lustre: Skipped 12 previous similar messages [ 3056.604982] Lustre: server umount lustre-MDT0000 complete [ 3059.681149] LustreError: 106987:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.700946] LustreError: 106987:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 3060.129204] LustreError: 103668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132120 with bad export cookie 2994774071464023832 [ 3060.131044] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3060.143183] LustreError: 103668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3060.520681] Lustre: server umount lustre-MDT0001 complete [ 3075.260681] Lustre: server umount lustre-OST0000 complete [ 3089.634787] Lustre: server umount lustre-OST0001 complete [ 3095.476313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3105.341372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3114.515253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3125.444643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3148.985257] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 3159.041704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.059616] Lustre: lustre-MDT0000: reset Object Index mappings [ 3159.632615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.636617] Lustre: Skipped 2 previous similar messages [ 3164.228681] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3173.613525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.628995] Lustre: lustre-MDT0001: reset Object Index mappings [ 3179.151273] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3182.116603] Lustre: 115856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3189.262469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3196.985956] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3198.563506] LustreError: 116209:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.585065] LustreError: 116209:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3198.604396] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3205.175368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3206.035209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3212.278502] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3212.291132] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3213.469417] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3221.981635] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3225.892572] Lustre: 117731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3277.498677] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:12:16 (1789132336) [ 3304.430808] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.445705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.453791] Lustre: Skipped 6 previous similar messages [ 3304.462105] Lustre: Skipped 5 previous similar messages [ 3308.727753] Lustre: server umount lustre-MDT0000 complete [ 3311.786705] LustreError: 114686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132372 with bad export cookie 2994774071464070928 [ 3311.791390] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3311.797048] LustreError: 114686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3318.272383] Lustre: server umount lustre-MDT0001 complete [ 3328.230228] Lustre: server umount lustre-OST0000 complete [ 3338.006889] Lustre: server umount lustre-OST0001 complete [ 3345.564607] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 3357.595872] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 3394.160690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.399223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3394.430270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3394.529762] Lustre: lustre-MDT0000: new disk, initializing [ 3394.602535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3394.610415] Lustre: Skipped 3 previous similar messages [ 3394.629400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3399.138367] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3406.152511] Lustre: server umount lustre-MDT0000 complete [ 3415.666795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3415.888438] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3415.909137] Lustre: 122003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3415.935270] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3420.499342] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3425.717610] Lustre: server umount lustre-MDT0000 complete [ 3434.459973] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:14:52 (1789132492) [ 3436.261023] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3438.195378] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:14:56 (1789132496) [ 3457.423378] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 3468.795290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.000314] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3481.459592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.645851] Lustre: 124053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3481.713424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3481.718154] Lustre: Skipped 1 previous similar message [ 3481.796983] Lustre: lustre-MDT0001: new disk, initializing [ 3481.929633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3481.936909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3485.945717] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3490.735404] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3497.185302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3497.477681] Lustre: lustre-OST0000: new disk, initializing [ 3497.489495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3497.499921] Lustre: 125628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3499.302833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3499.310061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3499.413732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3502.729711] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3512.420719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3512.625892] Lustre: lustre-OST0001: new disk, initializing [ 3512.632258] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3512.638510] Lustre: 126499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3513.956358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3513.971680] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3514.116771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3519.195565] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3527.936493] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3531.003067] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3902.826590] Lustre: 132908:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3902.832256] Lustre: 132908:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3902.837697] Lustre: 132908:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3902.845599] Lustre: 132908:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3902.851572] Lustre: 132908:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3902.856546] Lustre: 132908:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4218.196614] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:27:55 (1789133275) [ 4234.223070] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.240605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4234.260630] Lustre: Skipped 5 previous similar messages [ 4234.272086] Lustre: Skipped 7 previous similar messages [ 4240.346841] Lustre: server umount lustre-MDT0000 complete [ 4244.250318] LustreError: 124791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133305 with bad export cookie 2994774071464139493 [ 4244.254290] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4244.261851] LustreError: 124791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4244.448851] LustreError: 127313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.451934] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.471990] LustreError: 127313:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 4244.521426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4244.526751] Lustre: Skipped 5 previous similar messages [ 4250.967032] Lustre: server umount lustre-MDT0001 complete [ 4261.289757] Lustre: server umount lustre-OST0000 complete [ 4271.616610] Lustre: server umount lustre-OST0001 complete [ 4278.554442] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 4288.472107] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 4325.467479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.711262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4325.737241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4325.825788] Lustre: lustre-MDT0000: new disk, initializing [ 4325.918301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.923236] Lustre: Skipped 5 previous similar messages [ 4325.948518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4331.249957] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4341.426166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.521947] Lustre: 135967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4341.560225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4341.562976] Lustre: Skipped 1 previous similar message [ 4341.611224] Lustre: lustre-MDT0001: new disk, initializing [ 4341.752520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4341.759447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4345.939535] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4355.188462] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4361.233775] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4368.543435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4368.885700] Lustre: lustre-OST0000: new disk, initializing [ 4368.896353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4368.907739] Lustre: 137935:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4369.026845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.032705] Lustre: Skipped 1 previous similar message [ 4370.801777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4370.806785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4371.007623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4376.132468] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4388.450746] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4398.434890] Lustre: lustre-MDD0000: changelog on [ 4401.615953] Lustre: lustre-MDD0001: changelog on [ 6417.348839] Lustre: lustre-MDD0001: changelog off [ 6419.659792] Lustre: 137131:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6420.589131] Lustre: lustre-MDD0000: changelog off [ 6429.152633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6429.155054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6429.167723] LustreError: Skipped 1 previous similar message [ 6429.171690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6429.191168] Lustre: Skipped 2 previous similar messages [ 6429.202288] Lustre: Skipped 2 previous similar messages [ 6434.131760] Lustre: server umount lustre-OST0000 complete [ 6439.393366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.407985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6439.418765] Lustre: Skipped 1 previous similar message [ 6444.516038] LustreError: 135979:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.532025] LustreError: 135979:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6444.940660] Lustre: server umount lustre-MDT0000 complete [ 6448.615943] LustreError: 135960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135509 with bad export cookie 2994774071464960495 [ 6448.624637] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6448.629755] LustreError: 135960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6449.147901] Lustre: server umount lustre-MDT0001 complete [ 6460.504705] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 6464.605604] Key type lgssc unregistered [ 6464.985785] LNet: 147108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6465.011406] LNetError: 147108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6465.040792] LNet: Removed LNI 192.168.204.122@tcp [ 6466.211154] Key type .llcrypt unregistered [ 6466.213634] Key type ._llcrypt unregistered [ 6493.928698] Key type ._llcrypt registered [ 6493.936620] Key type .llcrypt registered [ 6494.140370] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 6517.480330] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 6520.104376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6520.159733] alg: No test for adler32 (adler32-zlib) [ 6521.532532] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 6522.010892] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 6523.887179] Key type lgssc registered [ 6525.772566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6566.549152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6566.619913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.878980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6567.906231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6567.963082] Lustre: lustre-MDT0000: new disk, initializing [ 6568.058678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6568.090778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6573.707842] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6585.984739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.139903] Lustre: 150312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6586.202559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6586.206559] Lustre: Skipped 1 previous similar message [ 6586.308395] Lustre: lustre-MDT0001: new disk, initializing [ 6586.382269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6586.409794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6586.425761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6591.159535] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6602.082649] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6609.644551] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6618.168746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6618.481060] Lustre: lustre-OST0000: new disk, initializing [ 6618.484547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6618.489691] Lustre: 152285:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6618.578143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6624.889661] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6626.382717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6626.397645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6626.658633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6635.330649] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6641.163833] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_module obdecho/obdecho [ 6642.909934] LustreError: 153287:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6642.924447] LustreError: 153287:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6644.495511] LustreError: 153383:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6657.507065] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6657.513611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6657.516387] Lustre: Skipped 2 previous similar messages [ 6657.529948] Lustre: Skipped 2 previous similar messages [ 6662.630076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6662.641707] Lustre: Skipped 2 previous similar messages [ 6663.809500] Lustre: server umount lustre-MDT0000 complete [ 6668.013305] LustreError: 150307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135728 with bad export cookie 16249475915911328313 [ 6668.033923] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6668.464361] Lustre: server umount lustre-MDT0001 complete [ 6683.048822] Lustre: server umount lustre-OST0000 complete [ 6706.082622] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 6718.322124] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 6760.333448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6760.709626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6760.775360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6760.911612] Lustre: lustre-MDT0000: new disk, initializing [ 6761.016649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6761.029685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6765.389167] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6776.933900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6777.081751] Lustre: 157099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6777.135286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6777.140681] Lustre: Skipped 1 previous similar message [ 6777.185944] Lustre: lustre-MDT0001: new disk, initializing [ 6777.365568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6777.442068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6777.465088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6782.206613] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6793.912868] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6801.201412] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6809.103288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6809.618904] Lustre: lustre-OST0000: new disk, initializing [ 6809.624733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6809.634599] Lustre: 159068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6809.748122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.293262] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6818.406938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6818.431053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6818.560166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6827.073581] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6833.059963] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6833.326820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6838.339835] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6838.695292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6844.394637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6844.410381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6844.413729] Lustre: Skipped 1 previous similar message [ 6849.523684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6849.533234] Lustre: Skipped 2 previous similar messages [ 6854.636860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6854.641455] Lustre: Skipped 1 previous similar message [ 6855.135118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6855.399938] Lustre: server umount lustre-OST0000 complete [ 6864.353878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6864.359114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.369805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6866.940718] Lustre: server umount lustre-MDT0000 complete [ 6869.995589] LustreError: 157113:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.010433] LustreError: 157113:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6871.950325] LustreError: 159067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135932 with bad export cookie 16249475915911329272 [ 6871.967308] LustreError: 159067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6871.967638] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6872.460244] Lustre: server umount lustre-MDT0001 complete [ 6877.458703] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:12:15 (1789135935) [ 6879.562100] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6881.800668] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:12:20 (1789135940) [ 6889.701601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.177493] LustreError: 161482:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6890.267717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.909936] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6895.592232] LustreError: 161483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6900.705923] LustreError: 161482:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6904.106589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.549600] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6919.048503] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6926.133837] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6934.851147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6935.524034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6935.531936] Lustre: Skipped 1 previous similar message [ 6942.316624] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6950.180649] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6956.341858] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.22@tcp (stopping) [ 6956.350822] Lustre: Skipped 1 previous similar message [ 6960.097469] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.114700] Lustre: Skipped 2 previous similar messages [ 6962.523203] Lustre: server umount lustre-MDT0000 complete [ 6965.222420] LustreError: 162211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6965.256978] LustreError: 162211:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6966.588649] LustreError: 164065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136027 with bad export cookie 16249475915911330154 [ 6966.602378] LustreError: 164065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6966.604638] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6970.345157] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.356349] Lustre: Skipped 1 previous similar message [ 6972.991544] Lustre: server umount lustre-MDT0001 complete [ 6980.064316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6995.151321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.440883] LustreError: 165327:0:(llog.c:1655:llog_backup()) MGC192.168.204.122@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6995.454768] Lustre: 165327:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.122@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6996.961730] LustreError: 165332:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.991546] LustreError: 165332:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 7013.601478] LustreError: 165334:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7013.626990] LustreError: 165334:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 7048.173262] LustreError: 163316:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.195616] LustreError: 163316:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 25 previous similar messages [ 7056.863330] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7056.870876] Lustre: 165327:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.122@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7097.860677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7101.413291] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7102.952528] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7109.974817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7112.161101] LustreError: 148533:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000019735121 x1876044632165632/t0(0) o250->MGC192.168.204.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7112.179750] LustreError: 165332:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.190306] LustreError: 165332:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 42 previous similar messages [ 7117.685433] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7117.817428] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7127.367938] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7128.914519] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7134.584244] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7136.433772] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7158.757430] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7158.759301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7158.774429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7158.774438] Lustre: Skipped 7 previous similar messages [ 7158.789054] Lustre: Skipped 1 previous similar message [ 7159.024280] Lustre: server umount lustre-MDT0000 complete [ 7163.222768] LustreError: 161463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136224 with bad export cookie 16249475915911336608 [ 7163.239592] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7163.243771] LustreError: 161463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7163.774315] Lustre: server umount lustre-MDT0001 complete [ 7177.574700] Lustre: server umount lustre-OST0000 complete [ 7199.949263] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 7211.363937] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7250.987128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7251.332556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7251.368224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7251.454541] Lustre: lustre-MDT0000: new disk, initializing [ 7251.529144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7251.531604] Lustre: Skipped 1 previous similar message [ 7251.553483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7256.651396] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7267.202993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.338809] Lustre: 170433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7267.766310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7272.903051] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7283.446656] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7289.955832] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7297.225521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7297.565128] Lustre: lustre-OST0000: new disk, initializing [ 7297.573136] Lustre: Skipped 1 previous similar message [ 7297.585427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7297.594121] Lustre: Skipped 2 previous similar messages [ 7297.604985] Lustre: 172404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7303.798643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7303.808301] Lustre: Skipped 1 previous similar message [ 7303.816905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7303.928873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7305.016482] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7315.195779] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7319.949381] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7320.150445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7325.248416] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7325.428840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7329.760473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7329.768128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.769014] LustreError: Skipped 1 previous similar message [ 7329.774570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7329.780525] Lustre: Skipped 2 previous similar messages [ 7329.792664] Lustre: Skipped 3 previous similar messages [ 7333.575678] Lustre: server umount lustre-OST0000 complete [ 7339.488541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7345.122309] LustreError: 170445:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.153852] LustreError: 170445:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7345.571480] Lustre: server umount lustre-MDT0000 complete [ 7350.260814] LustreError: 170426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136411 with bad export cookie 16249475915911353436 [ 7350.268531] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7350.269241] LustreError: 170426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7350.749035] Lustre: server umount lustre-MDT0001 complete [ 7360.921908] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:20:18 (1789136418) [ 7367.769579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.044748] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7381.791176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7387.224839] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7395.309573] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7401.463152] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7408.489734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7415.566938] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7423.571211] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7437.120621] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:21:35 (1789136495) [ 7441.889697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7441.902703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.916979] Lustre: Skipped 2 previous similar messages [ 7447.714336] Lustre: server umount lustre-OST0000 complete [ 7458.320911] Lustre: server umount lustre-MDT0000 complete [ 7462.332818] LustreError: 174797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136523 with bad export cookie 16249475915911354318 [ 7462.335978] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7462.342356] LustreError: 174797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7462.721893] Lustre: server umount lustre-MDT0001 complete [ 7474.242572] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 7478.039793] Key type lgssc unregistered [ 7478.489728] LNet: 178548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7478.500846] LNetError: 178548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7478.531380] LNet: Removed LNI 192.168.204.122@tcp [ 7479.463508] Key type .llcrypt unregistered [ 7479.466892] Key type ._llcrypt unregistered [ 7499.752208] Key type ._llcrypt registered [ 7499.756266] Key type .llcrypt registered [ 7499.899873] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7501.001076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7501.043800] alg: No test for adler32 (adler32-zlib) [ 7502.127814] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 7502.392640] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 7504.119195] Key type lgssc registered [ 7504.993207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7512.829549] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7512.871150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7528.415539] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7533.535391] LustreError: 179795:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.122@tcp: failed processing log, type 4: rc = -110 [ 7538.657420] LustreError: 179795:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7538.692421] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7538.704714] LustreError: MGC192.168.204.122@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7538.722904] LustreError: 179795:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7538.743727] LustreError: 179795:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7538.749671] LustreError: 179795:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7538.815716] Lustre: server umount lustre-OST0000 complete [ 7538.822766] LustreError: 179795:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7547.605682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7549.083591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7552.873611] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7558.626172] LustreError: 180294:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7558.642251] LustreError: 180294:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7560.917182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7561.485470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7565.261762] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7572.143581] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7579.301413] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7586.478574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7586.753555] LustreError: 182111:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7586.763961] Lustre: 182111:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.122@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7587.034527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7593.988402] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7600.922770] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7603.687403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.690481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7603.704837] Lustre: Skipped 2 previous similar messages [ 7603.715126] Lustre: Skipped 1 previous similar message [ 7606.821177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7606.824133] Lustre: Skipped 2 previous similar messages [ 7609.248436] Lustre: server umount lustre-MDT0000 complete [ 7611.875819] LustreError: 180294:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7611.891908] LustreError: 180294:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7612.714621] LustreError: 180274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136673 with bad export cookie 9660511022977936772 [ 7612.717693] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7612.724463] LustreError: 180274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7612.913306] LustreError: 181024:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e08430e1 x1876045660713600/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7613.100350] Lustre: server umount lustre-MDT0001 complete [ 7626.580657] Lustre: server umount lustre-OST0000 complete [ 7635.720671] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 7645.677756] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7681.110743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7681.334750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7681.378626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7681.478937] Lustre: lustre-MDT0000: new disk, initializing [ 7681.553420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.571693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7685.681935] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7695.475338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.579219] Lustre: 185632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7695.646973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7695.654181] Lustre: Skipped 1 previous similar message [ 7695.717154] Lustre: lustre-MDT0001: new disk, initializing [ 7695.802113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7695.868098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7695.897411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7700.224261] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7709.981439] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7715.885136] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7722.513350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7722.832526] Lustre: lustre-OST0000: new disk, initializing [ 7722.843066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7722.853748] Lustre: 187604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7722.968806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.697849] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7729.706276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7729.724191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7729.934671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7739.384745] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7745.608923] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7745.964741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7751.753658] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7752.078956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7755.745990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7755.756695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.773814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7755.780066] Lustre: Skipped 1 previous similar message [ 7755.795319] Lustre: Skipped 1 previous similar message [ 7760.599631] Lustre: server umount lustre-OST0000 complete [ 7765.989690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.992550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7766.012036] Lustre: Skipped 1 previous similar message [ 7770.773112] Lustre: server umount lustre-MDT0000 complete [ 7771.104839] LustreError: 185645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.118188] LustreError: 185645:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7774.303706] LustreError: 186428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136835 with bad export cookie 9660511022977937493 [ 7774.308976] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7774.311727] LustreError: 186428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7774.551373] Lustre: server umount lustre-MDT0001 complete [ 7783.229042] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:27:21 (1789136841) [ 7801.571620] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 7804.964441] Key type lgssc unregistered [ 7805.351249] LNet: 190552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7805.359782] LNetError: 190552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7805.400115] LNet: Removed LNI 192.168.204.122@tcp [ 7806.594988] Key type .llcrypt unregistered [ 7806.599211] Key type ._llcrypt unregistered [ 7826.296291] Key type ._llcrypt registered [ 7826.298785] Key type .llcrypt registered [ 7826.453425] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7827.410522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7827.427135] alg: No test for adler32 (adler32-zlib) [ 7828.537933] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 7828.743552] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 7830.407227] Key type lgssc registered [ 7831.889443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7839.099517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7839.139681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7840.519431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7844.899894] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7850.980052] LustreError: 191775:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7850.988731] LustreError: 191775:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7852.709758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7853.108240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7857.524635] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7864.560666] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7870.737922] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7880.178589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7880.606203] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7880.610756] LustreError: 193639:0:(llog.c:1696:llog_backup()) MGC192.168.204.122@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7880.616990] Lustre: 193639:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.122@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7880.757103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.496297] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7896.290102] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7899.112478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7899.117102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7899.130764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7902.693610] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.698353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7902.714627] Lustre: Skipped 2 previous similar messages [ 7904.540607] Lustre: server umount lustre-MDT0000 complete [ 7907.822809] LustreError: 191771:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.848603] LustreError: 191771:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7908.393223] LustreError: 191753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136969 with bad export cookie 4649874541142666536 [ 7908.402192] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7908.404572] LustreError: 191753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7908.711421] Lustre: server umount lustre-MDT0001 complete [ 7922.385186] Lustre: server umount lustre-OST0000 complete [ 7932.420368] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 7942.719469] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7978.170668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7978.361411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7978.400953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7978.474449] Lustre: lustre-MDT0000: new disk, initializing [ 7978.539077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7978.555838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7982.634500] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7992.910087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7993.034129] Lustre: 197165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7993.088174] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7993.090022] Lustre: Skipped 1 previous similar message [ 7993.093089] LustreError: 197946:0:(llog.c:1696:llog_backup()) MGC192.168.204.122@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7993.098451] LustreError: 197946:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7993.102309] Lustre: 197946:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.122@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7993.120150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7993.124033] Lustre: Skipped 1 previous similar message [ 7993.192891] Lustre: lustre-MDT0001: new disk, initializing [ 7993.269741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7993.298638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7993.309088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7998.223537] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8006.553399] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8011.707663] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8018.328632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8018.602607] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8018.605317] Lustre: Skipped 1 previous similar message [ 8018.609309] LustreError: 199109:0:(llog.c:1696:llog_backup()) MGC192.168.204.122@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8018.616701] LustreError: 199109:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8018.625361] Lustre: 199109:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.122@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8018.631642] Lustre: 199109:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8018.650230] Lustre: lustre-OST0000: new disk, initializing [ 8018.658482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8018.668954] Lustre: 199132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8018.774574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.280783] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8028.209947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8028.216935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8028.361721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8034.584765] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8040.180235] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8040.399756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8045.039534] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8045.265599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8047.014914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8047.016109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8047.029617] Lustre: Skipped 2 previous similar messages [ 8052.199910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8052.207431] Lustre: Skipped 2 previous similar messages [ 8053.122715] Lustre: server umount lustre-OST0000 complete [ 8057.317094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8057.327552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8057.342610] Lustre: Skipped 1 previous similar message [ 8057.359296] Lustre: Skipped 1 previous similar message [ 8062.873539] Lustre: server umount lustre-MDT0000 complete [ 8066.823304] LustreError: 197158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137127 with bad export cookie 4649874541142667250 [ 8066.837231] LustreError: 197158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8066.838805] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8067.282288] Lustre: server umount lustre-MDT0001 complete [ 8076.611560] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:32:14 (1789137134) [ 8093.959704] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 8103.884239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.318378] LustreError: 202258:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.376913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8109.199498] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8109.551583] LustreError: 202259:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.663273] LustreError: 202258:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.745710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.803977] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8125.736508] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8133.093768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8140.821829] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8150.135201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8150.396232] Lustre: lustre-OST0001: new disk, initializing [ 8150.408469] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8150.415324] Lustre: 204566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8150.574305] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8150.587990] Lustre: Skipped 2 previous similar messages [ 8156.847157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8156.873725] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8156.995519] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8157.513752] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8168.573274] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8175.312986] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8177.304649] Lustre: DEBUG MARKER: ADD OST3 [ 8182.440625] loop: module loaded [ 8182.509463] loop0: detected capacity change from 0 to 204800000 [ 8182.796433] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8182.809428] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8182.833859] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8190.222673] Lustre: DEBUG MARKER: STOP OST3 [ 8190.632514] loop0: detected capacity change from 0 to 204800000 [ 8197.320272] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8197.560378] Lustre: lustre-OST0003: new disk, initializing [ 8197.568205] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8197.579638] Lustre: 206626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8199.664957] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8199.799904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8199.813514] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8199.850258] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8200.553552] Lustre: Failing over lustre-OST0003 [ 8200.745494] Lustre: server umount lustre-OST0003 complete [ 8201.695270] LustreError: 206797:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8204.539912] LustreError: 203745:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8204.554906] LustreError: 203745:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8204.779071] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8204.796987] Lustre: Skipped 1 previous similar message [ 8205.042794] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8213.001664] Lustre: DEBUG MARKER: START OST3 again [ 8219.712531] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8220.117668] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8220.125700] Lustre: Skipped 1 previous similar message [ 8220.140774] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8221.622306] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8222.157911] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8222.159333] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8227.034500] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8248.902429] Lustre: Failing over lustre-OST0003 [ 8249.047727] Lustre: server umount lustre-OST0003 complete [ 8250.876415] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.890765] LustreError: 203745:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8250.908935] LustreError: 203745:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 8266.720986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8266.726107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8266.736474] Lustre: Skipped 2 previous similar messages [ 8271.336782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8271.354089] Lustre: Skipped 3 previous similar messages [ 8272.863829] Lustre: server umount lustre-MDT0000 complete [ 8276.134258] LustreError: 202240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137337 with bad export cookie 4649874541142668132 [ 8276.137065] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8276.148396] LustreError: 202240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8282.586426] Lustre: server umount lustre-MDT0001 complete [ 8287.312537] Lustre: server umount lustre-OST0000 complete [ 8291.434580] Lustre: server umount lustre-OST0001 complete [ 8297.988729] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 8307.228082] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 8349.314794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8349.639638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8349.679651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8349.806437] Lustre: lustre-MDT0000: new disk, initializing [ 8349.917373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8349.945506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8356.006552] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8368.738331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8368.835327] Lustre: 212030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8368.887313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8368.890330] Lustre: Skipped 1 previous similar message [ 8368.966292] Lustre: lustre-MDT0001: new disk, initializing [ 8369.171377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8369.202378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8373.244071] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8382.338256] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8386.821216] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8392.978054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8393.231328] Lustre: 213997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8395.130858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8395.317583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8399.530764] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8407.656684] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8412.242435] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8412.394374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8415.951248] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8416.191474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8420.838642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8420.839469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.840520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8420.840526] Lustre: Skipped 8 previous similar messages [ 8420.845675] LustreError: Skipped 1 previous similar message [ 8420.882276] Lustre: Skipped 4 previous similar messages [ 8424.523619] Lustre: server umount lustre-OST0000 complete [ 8430.562789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8435.542372] Lustre: server umount lustre-MDT0000 complete [ 8436.195229] LustreError: 212041:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8436.230147] LustreError: 212041:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 8439.637627] LustreError: 212023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137500 with bad export cookie 4649874541142670435 [ 8439.639034] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8439.649570] LustreError: 212023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8439.997631] Lustre: server umount lustre-MDT0001 complete [ 8443.025738] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:38:21 (1789137501) [ 8461.004545] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 8471.877572] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 8512.140596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8512.430344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8512.491443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8512.497202] Lustre: Skipped 1 previous similar message [ 8512.599581] Lustre: lustre-MDT0000: new disk, initializing [ 8512.606150] Lustre: Skipped 1 previous similar message [ 8512.747339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8512.762432] Lustre: Skipped 1 previous similar message [ 8517.873557] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8529.531479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8529.682722] Lustre: 218606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8529.907804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8535.920637] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8546.644113] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8552.949782] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8559.931215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8560.323855] Lustre: 220580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8561.792574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8566.272573] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8576.614987] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8581.956164] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8582.221920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8587.169300] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8587.461566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8592.355741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8592.358503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.369217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8592.377778] Lustre: Skipped 3 previous similar messages [ 8592.402426] Lustre: Skipped 5 previous similar messages [ 8604.127204] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8604.462832] Lustre: server umount lustre-OST0000 complete [ 8611.808819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8615.337431] Lustre: server umount lustre-MDT0000 complete [ 8617.964218] LustreError: 218614:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8617.983551] LustreError: 218614:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8619.211630] LustreError: 218601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137680 with bad export cookie 4649874541142671317 [ 8619.227512] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8619.232140] LustreError: 218601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8619.591062] Lustre: server umount lustre-MDT0001 complete [ 8629.459430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8630.029880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8630.036688] Lustre: Skipped 5 previous similar messages [ 8635.379503] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8644.585385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8649.961817] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8657.551305] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8663.581339] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8671.495368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8678.637973] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8686.794252] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8699.875814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8699.876856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8699.914406] Lustre: Skipped 8 previous similar messages [ 8703.430775] Lustre: server umount lustre-OST0000 complete [ 8713.999820] Lustre: server umount lustre-MDT0000 complete [ 8718.094702] LustreError: 222777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137778 with bad export cookie 4649874541142672199 [ 8718.098135] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8718.104497] LustreError: 222777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8718.503626] Lustre: server umount lustre-MDT0001 complete [ 8827.576619] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 8830.920424] Key type lgssc unregistered [ 8831.361148] LNet: 226467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8831.369790] LNetError: 226467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8832.434534] LNet: Removed LNI 192.168.204.122@tcp [ 8833.477177] Key type .llcrypt unregistered [ 8833.482688] Key type ._llcrypt unregistered [ 8860.471754] Key type ._llcrypt registered [ 8860.474493] Key type .llcrypt registered [ 8860.613475] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 8877.455508] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 8878.249733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8878.258173] alg: No test for adler32 (adler32-zlib) [ 8879.288399] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 8879.540135] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 8881.207369] Key type lgssc registered [ 8882.524620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8917.810305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8917.845498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.161349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8919.227803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8919.324995] Lustre: lustre-MDT0000: new disk, initializing [ 8919.407353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.440881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8924.854828] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8936.646471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.774987] Lustre: 229823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8936.830895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8936.835254] Lustre: Skipped 1 previous similar message [ 8936.968168] Lustre: lustre-MDT0001: new disk, initializing [ 8937.148184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8937.261353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8937.285962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8943.610058] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8953.820836] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8959.980400] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8967.658994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8968.014960] Lustre: lustre-OST0000: new disk, initializing [ 8968.020974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8968.034156] Lustre: 231789:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8968.180737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8974.995529] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8976.518492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8976.534234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8976.631955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8985.397861] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8990.477460] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8990.740289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8995.856060] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8996.161746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9002.465460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9002.474765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.480275] LustreError: Skipped 1 previous similar message [ 9002.483497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9002.488303] Lustre: Skipped 1 previous similar message [ 9002.498708] Lustre: Skipped 1 previous similar message [ 9004.221735] Lustre: server umount lustre-OST0000 complete [ 9009.121096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9009.129357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9009.142249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9012.706696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9012.711804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9014.314049] Lustre: server umount lustre-MDT0000 complete [ 9017.898423] LustreError: 229815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138078 with bad export cookie 16128656781472255149 [ 9017.903843] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9017.913413] LustreError: 229815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9018.295639] Lustre: server umount lustre-MDT0001 complete [ 9039.873985] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 9051.204270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9051.577745] LustreError: 234726:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9051.594970] LustreError: 234726:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9051.707328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9056.737532] LustreError: 234727:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9056.853492] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9061.872205] LustreError: 234726:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.002214] LustreError: 234727:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.384266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9067.883749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9073.021036] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9076.346774] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9083.805733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9084.339599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9091.350376] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9100.260666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9100.523451] Lustre: lustre-OST0001: new disk, initializing [ 9100.527641] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9100.536274] Lustre: 237034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9100.687212] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9103.415253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9103.448708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9103.688969] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9109.040500] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9121.694897] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9128.807133] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9143.609768] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:50:02 (1789138202) [ 9149.928392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9149.928893] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9149.942808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9149.942817] Lustre: Skipped 1 previous similar message [ 9149.979825] Lustre: Skipped 3 previous similar messages [ 9152.128899] Lustre: server umount lustre-MDT0000 complete [ 9155.047857] LustreError: 235454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9155.091362] LustreError: 235454:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9156.115155] LustreError: 237802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138216 with bad export cookie 16128656781472256031 [ 9156.132161] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9156.586944] Lustre: server umount lustre-MDT0001 complete [ 9171.727468] Lustre: server umount lustre-OST0000 complete [ 9185.761272] Lustre: server umount lustre-OST0001 complete [ 9193.876505] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 9204.319175] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 9245.130195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9245.335593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9245.380961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9245.459278] Lustre: lustre-MDT0000: new disk, initializing [ 9245.521084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9245.549207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9250.899350] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9262.186770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.342823] Lustre: 240981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9262.400687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9262.404991] Lustre: Skipped 1 previous similar message [ 9262.496850] Lustre: lustre-MDT0001: new disk, initializing [ 9262.714095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9262.736799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9268.729550] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9279.250231] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9285.667957] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9293.247530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9293.625888] Lustre: lustre-OST0000: new disk, initializing [ 9293.630543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9293.641229] Lustre: 242952:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9294.993414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9295.007942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9295.178821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9300.790515] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9312.718539] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9319.685877] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9320.033817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9326.317382] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9326.620635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9331.174654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9331.181458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9331.185505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9331.185516] Lustre: Skipped 3 previous similar messages [ 9331.200861] Lustre: Skipped 1 previous similar message [ 9331.215815] LustreError: Skipped 1 previous similar message [ 9335.435066] Lustre: server umount lustre-OST0000 complete [ 9341.412259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9341.419058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9341.433029] Lustre: Skipped 1 previous similar message [ 9341.443215] Lustre: Skipped 2 previous similar messages [ 9346.404541] Lustre: server umount lustre-MDT0000 complete [ 9346.532571] LustreError: 240988:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9346.563066] LustreError: 240988:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9350.908544] LustreError: 240974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138411 with bad export cookie 16128656781472258166 [ 9350.919117] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9350.922405] LustreError: 240974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9351.449638] Lustre: server umount lustre-MDT0001 complete [ 9362.085977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.755643] LustreError: 245172:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9362.848568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9362.854143] Lustre: Skipped 2 previous similar messages [ 9368.571543] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9379.045698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9385.159344] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9394.130615] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9401.293562] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9410.084741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9419.375458] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9430.275812] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9578.976774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9578.983488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9579.001845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9582.388539] Lustre: server umount lustre-OST0000 complete [ 9589.729119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9589.748913] LustreError: Skipped 1 previous similar message [ 9593.242903] Lustre: server umount lustre-MDT0000 complete [ 9596.392771] LustreError: 245172:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9596.403804] LustreError: 245172:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 9597.128139] LustreError: 245153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138657 with bad export cookie 16128656781472259048 [ 9597.129200] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9597.143971] LustreError: 245153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9597.616804] Lustre: server umount lustre-MDT0001 complete [ 9700.321504] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 9703.539907] Key type lgssc unregistered [ 9703.872329] LNet: 248787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9703.880512] LNetError: 248787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9703.897863] LNet: Removed LNI 192.168.204.122@tcp [ 9705.025176] Key type .llcrypt unregistered [ 9705.027172] Key type ._llcrypt unregistered [ 9732.388322] Key type ._llcrypt registered [ 9732.390383] Key type .llcrypt registered [ 9732.566943] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 9750.727304] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 9752.023910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9752.075552] alg: No test for adler32 (adler32-zlib) [ 9753.188673] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [ 9753.467290] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 9755.162174] Key type lgssc registered [ 9756.692635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9806.010414] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 9818.066183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9818.104481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9819.400555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9819.435375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9819.522471] Lustre: lustre-MDT0000: new disk, initializing [ 9819.627477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9819.652916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9824.719461] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9835.434465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9835.545427] Lustre: 252873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9835.598781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9835.601765] Lustre: Skipped 1 previous similar message [ 9835.671994] Lustre: lustre-MDT0001: new disk, initializing [ 9835.768222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9835.829983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9835.842129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9841.595675] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9847.420237] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9854.484984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9854.919496] Lustre: lustre-OST0000: new disk, initializing [ 9854.925303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9854.942200] Lustre: 254501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9855.124281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9856.482900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9856.506562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9856.672105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9862.786264] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9875.760515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9876.016866] Lustre: lustre-OST0001: new disk, initializing [ 9876.033110] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9876.043372] Lustre: 255374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9876.233498] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9883.642993] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9885.913091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9885.936548] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9886.040722] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9896.816429] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9907.735532] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9922.026271] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9922.041610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9922.054334] Lustre: Skipped 1 previous similar message [ 9922.078515] Lustre: Skipped 3 previous similar messages [ 9927.152268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9927.158735] Lustre: Skipped 3 previous similar messages [ 9932.262859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9932.267735] Lustre: Skipped 2 previous similar messages [ 9934.303261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9934.559241] Lustre: server umount lustre-MDT0000 complete [ 9937.980564] LustreError: 252863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138998 with bad export cookie 10146205062140918746 [ 9937.984432] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9937.986522] LustreError: 252863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9938.190417] Lustre: server umount lustre-MDT0001 complete [ 9951.878874] Lustre: server umount lustre-OST0000 complete [ 9966.703366] Lustre: server umount lustre-OST0001 complete [ 9979.903114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9989.564805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9998.075726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10006.980760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10026.206404] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [10035.946946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.969857] Lustre: lustre-MDT0000: reset Object Index mappings [10036.507079] LustreError: 260390:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.520651] LustreError: 260390:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10036.597608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10040.852516] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10041.831515] LustreError: 260391:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10046.948138] LustreError: 260390:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10050.040939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10050.062390] Lustre: lustre-MDT0001: reset Object Index mappings [10050.558376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10055.887080] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10058.929275] Lustre: 261532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10065.118524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10073.175436] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10075.822525] LustreError: 261888:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10080.708852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10080.991715] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10081.001775] Lustre: Skipped 1 previous similar message [10082.046979] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10082.063973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10089.256460] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10097.407682] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10101.356924] Lustre: 263404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10106.853418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10106.860105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.873116] Lustre: Skipped 2 previous similar messages [10106.881196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10107.879683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.904695] Lustre: Skipped 1 previous similar message [10109.058727] Lustre: server umount lustre-MDT0000 complete [10113.011697] LustreError: 260391:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10113.034894] LustreError: 260391:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10113.509872] LustreError: 260370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139174 with bad export cookie 10146205062140925536 [10113.525748] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10113.535572] LustreError: 260370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10114.028205] Lustre: server umount lustre-MDT0001 complete [10127.466661] Lustre: server umount lustre-OST0000 complete [10141.905841] Lustre: server umount lustre-OST0001 complete [10156.609800] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:06:55 (1789139215) [10174.740423] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [10187.434595] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [10223.550764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10223.781889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10223.816500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10223.921170] Lustre: lustre-MDT0000: new disk, initializing [10224.025512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10224.045279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10228.648401] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10240.290779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10240.490364] Lustre: 267658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10240.558514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10240.564827] Lustre: Skipped 1 previous similar message [10240.717702] Lustre: lustre-MDT0001: new disk, initializing [10240.987168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10241.004513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10245.948665] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10254.760736] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10260.493386] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10268.600000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10268.941485] Lustre: lustre-OST0000: new disk, initializing [10268.950859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10268.960667] Lustre: 269625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10270.083893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10270.100800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10270.280742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10276.642129] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10285.750428] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10290.928938] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10291.137792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10295.913292] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10296.147039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10300.900154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10300.902328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10300.914685] LustreError: Skipped 1 previous similar message [10300.926583] Lustre: Skipped 1 previous similar message [10300.928170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10300.928177] Lustre: Skipped 4 previous similar messages [10304.484661] Lustre: server umount lustre-OST0000 complete [10311.138875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10311.145414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10311.156067] Lustre: Skipped 2 previous similar messages [10311.159748] Lustre: Skipped 1 previous similar message [10322.403878] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10322.588926] Lustre: server umount lustre-MDT0000 complete [10325.472728] LustreError: 270152:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.487789] LustreError: 270152:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10326.072938] LustreError: 267650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139386 with bad export cookie 10146205062140927629 [10326.081866] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10326.091686] LustreError: 267650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10326.469768] Lustre: server umount lustre-MDT0001 complete [10345.739793] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [10354.734633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10355.015062] LustreError: 272561:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10355.125289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10355.129505] Lustre: Skipped 2 previous similar messages [10359.620974] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10368.199146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10373.886110] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10377.341436] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10385.202419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10392.240852] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10400.294308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10400.490957] Lustre: lustre-OST0001: new disk, initializing [10400.494201] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10400.498814] Lustre: 274871:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10401.835339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10401.848405] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10401.985887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10406.658799] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10416.126520] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10419.557695] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10423.494020] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10423.512393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10429.923162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10429.937377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10429.961685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10429.970354] Lustre: Skipped 5 previous similar messages [10435.242721] Lustre: server umount lustre-MDT0000 complete [10438.117525] LustreError: 273289:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10438.133522] LustreError: 273289:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10438.728348] LustreError: 272541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139499 with bad export cookie 10146205062140928511 [10438.736273] LustreError: 272541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10438.736771] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10439.104665] Lustre: server umount lustre-MDT0001 complete [10452.553922] Lustre: server umount lustre-OST0000 complete [10466.216544] Lustre: server umount lustre-OST0001 complete [10482.918535] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [10494.983116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.760032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10495.771527] Lustre: Skipped 3 previous similar messages [10500.984730] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10505.193221] LustreError: 277925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10505.209707] LustreError: 277925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10508.781459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.633681] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10516.346512] Lustre: 279065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10522.576435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10529.451166] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10538.852428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10540.319158] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10540.333243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10544.991097] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10552.632504] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10557.107549] Lustre: 280943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10563.116868] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10563.210960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10572.935475] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:13:51 (1789139631) [10586.601641] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10586.616847] Lustre: Skipped 3 previous similar messages [10586.619781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10586.637545] Lustre: Skipped 4 previous similar messages [10591.200248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10592.907950] Lustre: server umount lustre-MDT0000 complete [10597.004192] LustreError: 277905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139657 with bad export cookie 10146205062140935056 [10597.033066] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10597.033618] LustreError: 277905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10603.752870] Lustre: server umount lustre-MDT0001 complete [10614.217415] Lustre: server umount lustre-OST0000 complete [10618.306907] Lustre: server umount lustre-OST0001 complete [10624.993470] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [10634.654149] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [10679.989203] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [10691.441324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10691.904620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10691.942065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10692.030680] Lustre: lustre-MDT0000: new disk, initializing [10692.128887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10696.025923] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10707.853054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.998993] Lustre: 285181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10708.384613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10713.899477] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10726.432585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10726.693715] Lustre: 286814:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10728.361760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10733.712965] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10744.529177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10744.763881] Lustre: 287686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10746.704134] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10746.719097] Lustre: Skipped 1 previous similar message [10746.870475] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10751.987300] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10762.218000] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10765.968790] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10765.973741] Lustre: Skipped 1 previous similar message [10779.924584] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:17:18 (1789139838) [10782.704015] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10782.709742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10782.727100] Lustre: Skipped 5 previous similar messages [10782.750519] Lustre: Skipped 13 previous similar messages [10784.729854] Lustre: server umount lustre-MDT0000 complete [10787.819441] LustreError: 286716:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.832077] LustreError: 286716:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [10788.713518] LustreError: 285174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139849 with bad export cookie 10146205062140942735 [10788.714932] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10788.741809] LustreError: 285174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10789.193502] Lustre: server umount lustre-MDT0001 complete [10803.828125] Lustre: server umount lustre-OST0000 complete [10817.884386] Lustre: server umount lustre-OST0001 complete [10827.905599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10828.496537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10828.508378] Lustre: Skipped 7 previous similar messages [10833.355832] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10842.211519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10847.222561] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10854.209364] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10859.948658] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10866.695982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10873.080308] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10880.453953] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10896.382542] loop1: detected capacity change from 0 to 204800000 [10896.754950] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10896.782236] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10896.822780] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10901.492823] loop1: detected capacity change from 0 to 204800000 [10907.404796] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10907.498273] LustreError: 289925:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12230: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10907.523283] LustreError: 289925:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12230 [10907.545451] LustreError: 289925:0:(mgs_handler.c:517:mgs_target_reg()) fs12230-MDT0001: Error checking target index: rc = -13 [10907.568156] LustreError: fs12230-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10907.580393] LustreError: 293659:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10907.586933] LustreError: 293659:0:(tgt_mount.c:2129:server_put_super()) no obd fs12230-MDT0001 [10907.593453] LustreError: 293659:0:(tgt_mount.c:116:server_deregister_mount()) fs12230-MDT0001 not registered [10907.635897] Lustre: server umount fs12230-MDT0001 complete [10907.638915] LustreError: 293659:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10916.547175] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10916.738882] Lustre: Setting parameter fs12230-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12230-MDT0001 [10916.799925] Lustre: srv-fs12230-MDT0001: No data found on store. Initialize space. [10916.803713] Lustre: Skipped 4 previous similar messages [10916.895700] Lustre: fs12230-MDT0001: new disk, initializing [10916.900810] Lustre: Skipped 3 previous similar messages [10917.025406] Lustre: srv-fs12230-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10917.037057] Lustre: 294127:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12230-MDT0001-mdtlov: can't create new llog: rc = -115 [10917.049882] LustreError: 294127:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12230-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10921.883361] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10929.445026] Lustre: server umount fs12230-MDT0001 complete [10943.108802] loop1: detected capacity change from 0 to 204800000 [10948.918529] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.576447] Lustre: 295693:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12230-MDT0001-mdtlov: can't create new llog: rc = -115 [10949.597086] LustreError: 295693:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12230-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10954.638931] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10955.248630] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10959.895394] Lustre: server umount fs12230-MDT0001 complete [10970.597714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10970.603080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10970.612744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10970.613534] Lustre: Skipped 4 previous similar messages [10970.636667] LustreError: Skipped 1 previous similar message [10976.621881] Lustre: server umount lustre-MDT0000 complete [10984.670900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.765737] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10989.762836] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10990.582199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10998.237941] loop1: detected capacity change from 0 to 204800000 [11003.784435] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.269460] Lustre: 298138:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12230-MDT0001-mdtlov: can't create new llog: rc = -115 [11004.277881] LustreError: 298138:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12230-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11008.418764] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11013.425691] Lustre: server umount fs12230-MDT0001 complete [11036.147546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11042.135626] Lustre: server umount lustre-OST0000 complete [11052.747175] Lustre: server umount lustre-MDT0000 complete [11055.013987] LustreError: 289934:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11055.027369] LustreError: 289934:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 107 previous similar messages [11056.179936] LustreError: 289919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140117 with bad export cookie 10146205062140945416 [11056.182760] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11056.196567] LustreError: 289919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11056.543521] Lustre: server umount lustre-MDT0001 complete [11069.415541] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [11073.041550] Key type lgssc unregistered [11073.433701] LNet: 300487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11073.439055] LNetError: 300487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11073.468359] LNet: Removed LNI 192.168.204.122@tcp [11074.538162] Key type .llcrypt unregistered [11074.539868] Key type ._llcrypt unregistered [11096.397724] Key type ._llcrypt registered [11096.406764] Key type .llcrypt registered [11096.568132] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [11097.740095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11097.838578] alg: No test for adler32 (adler32-zlib) [11099.018773] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [11099.300635] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [11101.000065] Key type lgssc registered [11102.307151] Lustre: Echo OBD driver; http://www.lustre.org/ [11110.833967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11110.850940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11112.394296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11117.255957] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11122.658101] LustreError: 301759:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11122.676543] LustreError: 301759:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11125.596749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11126.157176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11130.588630] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11138.730335] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11144.392582] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11152.514088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11153.016931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11158.120660] LustreError: 303590:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.738224] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11163.625933] LustreError: 303591:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11168.756706] LustreError: 303591:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11169.748064] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11175.951907] LustreError: 303589:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.978303] LustreError: 303589:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [11186.162751] LustreError: 303590:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11186.208276] LustreError: 303590:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11188.902336] loop1: detected capacity change from 0 to 204800000 [11189.135379] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11189.170509] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11189.195143] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11195.669801] loop1: detected capacity change from 0 to 204800000 [11202.286345] LustreError: 303820:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11202.313479] LustreError: 303820:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [11203.099639] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11203.299965] Lustre: fs26256-OST0000: new disk, initializing [11203.304859] Lustre: srv-fs26256-OST0000: No data found on store. Initialize space. [11203.312474] Lustre: 305497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11203.427869] Lustre: fs26256-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11210.019589] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11221.252328] Lustre: server umount fs26256-OST0000 complete [11237.355419] LustreError: 303820:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11237.377928] LustreError: 303820:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 19 previous similar messages [11237.398181] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11238.317033] loop1: detected capacity change from 0 to 204800000 [11245.367094] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11245.635659] Lustre: fs26256-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11252.725802] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11253.982431] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11264.644969] loop2: detected capacity change from 0 to 204800000 [11264.950113] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11264.959935] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11264.976128] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11270.389872] loop2: detected capacity change from 0 to 204800000 [11275.972815] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11276.057084] LustreError: fs26256-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11276.063821] LustreError: fs26256-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11276.072836] LustreError: 308540:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11276.077529] LustreError: 308540:0:(tgt_mount.c:2129:server_put_super()) no obd fs26256-OST0001 [11276.081221] LustreError: 308540:0:(tgt_mount.c:116:server_deregister_mount()) fs26256-OST0001 not registered [11276.107706] Lustre: server umount fs26256-OST0001 complete [11276.110163] LustreError: 308540:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11286.005361] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11286.274431] Lustre: fs26256-OST0001: new disk, initializing [11286.280703] Lustre: srv-fs26256-OST0001: No data found on store. Initialize space. [11286.287162] Lustre: 309090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11286.401587] Lustre: fs26256-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11292.616978] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11300.497427] Lustre: server umount fs26256-OST0000 complete [11303.914223] LustreError: 303591:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.929082] LustreError: 303591:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 38 previous similar messages [11309.577175] Lustre: server umount fs26256-OST0001 complete [11321.987586] loop1: detected capacity change from 0 to 204800000 [11328.315248] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11328.769576] Lustre: fs26256-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11335.895111] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11343.491191] loop2: detected capacity change from 0 to 204800000 [11349.459078] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11358.185300] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11368.626548] Lustre: server umount fs26256-OST0000 complete [11378.884064] Lustre: server umount fs26256-OST0001 complete [11390.974574] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11394.019268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11394.029086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11394.049403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11396.065431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11396.068579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11401.196717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11401.205378] Lustre: Skipped 2 previous similar messages [11406.304971] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11406.308170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11406.309276] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11406.309281] LustreError: Skipped 1 previous similar message [11406.345735] Lustre: Skipped 1 previous similar message [11406.538394] Lustre: server umount lustre-OST0000 complete [11411.428761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11411.432093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11411.449750] Lustre: Skipped 1 previous similar message [11411.464571] Lustre: Skipped 1 previous similar message [11416.866967] Lustre: server umount lustre-MDT0000 complete [11420.498865] LustreError: 301741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140481 with bad export cookie 9603278945193051581 [11420.506573] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11420.521868] LustreError: 301741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11421.327922] Lustre: server umount lustre-MDT0001 complete [11433.591789] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [11436.284112] Key type lgssc unregistered [11436.567963] LNet: 315018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11436.575511] LNetError: 315018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11436.599392] LNet: Removed LNI 192.168.204.122@tcp [11437.444273] Key type .llcrypt unregistered [11437.449375] Key type ._llcrypt unregistered [11472.038802] Key type ._llcrypt registered [11472.041692] Key type .llcrypt registered [11472.154206] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [11488.016371] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [11489.004764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11489.097376] alg: No test for adler32 (adler32-zlib) [11490.416899] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [11490.814741] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [11492.559197] Key type lgssc registered [11493.908732] Lustre: Echo OBD driver; http://www.lustre.org/ [11541.434669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11541.508193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11542.803727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11542.850644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11542.911228] Lustre: lustre-MDT0000: new disk, initializing [11542.985463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11543.018954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11548.956224] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11561.496300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11561.606420] Lustre: 318886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11561.713735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11561.717908] Lustre: Skipped 1 previous similar message [11561.810941] Lustre: lustre-MDT0001: new disk, initializing [11561.909419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11561.960212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11561.978309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11568.309504] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11578.516627] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11584.105662] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11591.034828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11591.301436] Lustre: lustre-OST0000: new disk, initializing [11591.304913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11591.312463] Lustre: 320855:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11591.422569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11599.158607] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11600.955405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11600.968928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11601.101477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11609.453429] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11611.606183] Lustre: Mounted lustre-client - version 2.17.57_133_g066c38c [11612.697646] Lustre: Unmounted lustre-client [11621.861501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11621.871581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11621.879853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11621.885214] Lustre: Skipped 1 previous similar message [11626.995945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11627.007761] Lustre: Skipped 2 previous similar messages [11627.608946] Lustre: server umount lustre-OST0000 complete [11633.633715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11633.640588] LustreError: Skipped 1 previous similar message [11633.643629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11633.661015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11637.216279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11637.218248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11638.654960] Lustre: server umount lustre-MDT0000 complete [11643.231747] LustreError: 318876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140704 with bad export cookie 2670308498349441850 [11643.247436] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11643.261543] LustreError: 318876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11644.161669] Lustre: server umount lustre-MDT0001 complete [11657.168428] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [11660.235560] Key type lgssc unregistered [11672.928637] LNet: 323017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11672.945199] LNetError: 323017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11740.521194] LNet: Removed LNI 192.168.204.122@tcp [11741.223345] Key type .llcrypt unregistered [11741.225291] Key type ._llcrypt unregistered [11760.224521] Key type ._llcrypt registered [11760.230387] Key type .llcrypt registered [11760.401244] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [11776.818774] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [11777.956496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11778.041129] alg: No test for adler32 (adler32-zlib) [11779.035319] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [11779.220794] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [11780.905758] Key type lgssc registered [11782.357737] Lustre: Echo OBD driver; http://www.lustre.org/ [11815.779504] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:34:33 (1789140873) [11821.956521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11821.973706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11823.170321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11823.194667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11823.259966] Lustre: lustre-MDT0000: new disk, initializing [11823.351864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11823.371958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11828.028991] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11839.765156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11839.857807] Lustre: 326241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11839.908934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11839.914154] Lustre: Skipped 1 previous similar message [11840.021559] Lustre: lustre-MDT0001: new disk, initializing [11840.118590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11840.215063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11840.222965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11845.740595] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11854.869535] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11860.642298] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11869.433049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11869.809950] Lustre: lustre-OST0000: new disk, initializing [11869.816624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11869.822879] Lustre: 328209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11870.025535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11879.044685] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11880.066732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11880.086976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11880.291978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11890.366371] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11911.141417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11911.142802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11911.154628] LustreError: Skipped 1 previous similar message [11911.156364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11911.186841] Lustre: Skipped 1 previous similar message [11913.650849] Lustre: server umount lustre-OST0000 complete [11921.400285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11921.412780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11921.432142] Lustre: Skipped 1 previous similar message [11921.437326] Lustre: Skipped 1 previous similar message [11924.221864] Lustre: server umount lustre-MDT0000 complete [11928.496246] LustreError: 326231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140989 with bad export cookie 13199821772777447289 [11928.501423] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11928.508507] LustreError: 326231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11929.117929] Lustre: server umount lustre-MDT0001 complete [11940.504705] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [11943.719468] Key type lgssc unregistered [11944.107860] LNet: 330121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11944.120360] LNetError: 330121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11944.139240] LNet: Removed LNI 192.168.204.122@tcp [11945.087234] Key type .llcrypt unregistered [11945.089741] Key type ._llcrypt unregistered [11966.171411] Key type ._llcrypt registered [11966.175498] Key type .llcrypt registered [11966.299816] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [11967.847971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11967.965957] alg: No test for adler32 (adler32-zlib) [11969.193146] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [11969.424689] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [11971.095171] Key type lgssc registered [11972.253704] Lustre: Echo OBD driver; http://www.lustre.org/ [11980.479211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11980.504535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11981.988568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11986.891592] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11991.525289] LustreError: 331396:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11991.537604] LustreError: 331396:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11994.200812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11994.602156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11998.121804] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12004.829101] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12011.453049] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12019.242432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12020.030684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12028.098744] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12036.819975] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12048.581833] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12050.519272] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12054.774774] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.22@tcp (stopping) [12056.034917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12056.051150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12056.069556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12057.060660] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12057.079354] Lustre: Skipped 1 previous similar message [12057.086422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12057.089077] Lustre: Skipped 2 previous similar messages [12059.873063] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.22@tcp (stopping) [12060.973702] Lustre: server umount lustre-MDT0000 complete [12062.176982] LustreError: 331396:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12062.200903] LustreError: 331396:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12065.009101] LustreError: 333856:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12065.225568] LustreError: 331376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141126 with bad export cookie 9029426179009930093 [12065.237173] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12065.240762] LustreError: 331376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12067.296535] LustreError: 331391:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.298344] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12067.333450] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12071.927573] Lustre: server umount lustre-MDT0001 complete [12076.123734] Lustre: server umount lustre-OST0000 complete [12087.490963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12092.619770] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12095.925717] Lustre: 335577:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.122@tcp [12098.962818] Lustre: server umount MGS complete [12108.719643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12109.296174] LustreError: 336149:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12109.316848] LustreError: 336149:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12109.439965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12114.265796] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12120.032926] LustreError: 336150:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12120.058558] LustreError: 336150:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12124.130598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12124.612351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12130.796913] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12139.755642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12140.033757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12145.775735] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12155.272508] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12157.255707] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12163.788282] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12165.708678] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12171.755427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12171.762588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12171.778094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12171.780454] Lustre: Skipped 2 previous similar messages [12173.214244] Lustre: server umount lustre-OST0000 complete [12179.936364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12179.959247] Lustre: Skipped 1 previous similar message [12183.954664] Lustre: server umount lustre-MDT0000 complete [12185.074780] LustreError: 337886:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12185.105228] LustreError: 337886:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12188.042415] LustreError: 336129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141248 with bad export cookie 9029426179009930933 [12188.052514] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12188.057872] LustreError: 336129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12197.569403] Lustre: server umount lustre-MDT0001 complete [12210.920366] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12214.439594] Key type lgssc unregistered [12214.803983] LNet: 339592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12214.821312] LNetError: 339592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12214.835223] LNet: Removed LNI 192.168.204.122@tcp [12215.942808] Key type .llcrypt unregistered [12215.950283] Key type ._llcrypt unregistered [12234.352347] Key type ._llcrypt registered [12234.356256] Key type .llcrypt registered [12234.486883] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [12255.197529] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [12257.188582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12257.230951] alg: No test for adler32 (adler32-zlib) [12258.734974] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [12259.304809] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [12260.991168] Key type lgssc registered [12262.262934] Lustre: Echo OBD driver; http://www.lustre.org/ [12301.823034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12301.864875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12303.198160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12303.240969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12303.340936] Lustre: lustre-MDT0000: new disk, initializing [12303.461242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12303.491977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12308.625467] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12319.779847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12319.908789] Lustre: 342759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12319.965628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12319.969584] Lustre: Skipped 1 previous similar message [12320.026718] Lustre: lustre-MDT0001: new disk, initializing [12320.122328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12320.155291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12320.160849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12324.861930] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12335.207556] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12341.444487] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12348.414938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12348.728431] Lustre: lustre-OST0000: new disk, initializing [12348.732703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12348.743351] Lustre: 344727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12348.878302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12355.188980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12355.217281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12355.377093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12355.860610] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12366.766225] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12372.119614] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12372.461638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12378.032500] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12378.293372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12381.157767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12381.165847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12381.173673] Lustre: Skipped 1 previous similar message [12381.183743] Lustre: Skipped 1 previous similar message [12386.275331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12386.285508] Lustre: Skipped 1 previous similar message [12387.166683] Lustre: server umount lustre-OST0000 complete [12396.512806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12396.535988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12396.542958] Lustre: Skipped 1 previous similar message [12398.546962] Lustre: server umount lustre-MDT0000 complete [12402.100431] LustreError: 342751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141462 with bad export cookie 1149955609997414251 [12402.101643] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12402.123967] LustreError: 342751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12402.603491] Lustre: server umount lustre-MDT0001 complete [12413.626221] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:44:31 (1789141471) [12434.399444] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [12446.009846] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [12489.052632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12489.537293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12489.587741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12489.680668] Lustre: lustre-MDT0000: new disk, initializing [12489.800785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12489.817803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12495.235903] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12506.880962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12507.065071] Lustre: 349490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12507.213068] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12507.222587] Lustre: Skipped 1 previous similar message [12507.334752] Lustre: lustre-MDT0001: new disk, initializing [12507.457347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12507.518847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12507.535857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12513.295751] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12525.447834] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12532.173715] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12539.837835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12540.210462] Lustre: lustre-OST0000: new disk, initializing [12540.213734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12540.224534] Lustre: 351462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12540.327860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12548.281759] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12549.265807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12549.280112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12549.402960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12560.319360] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12605.929432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12605.932181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12605.932876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12605.951889] Lustre: Skipped 2 previous similar messages [12606.029057] Lustre: Skipped 1 previous similar message [12608.039739] Lustre: server umount lustre-OST0000 complete [12615.138459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12615.151739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12615.176111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12619.497705] Lustre: server umount lustre-MDT0000 complete [12621.293834] LustreError: 349502:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12621.324174] LustreError: 349502:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12623.786892] LustreError: 349481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141684 with bad export cookie 1149955609997415133 [12623.805465] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12623.813491] LustreError: 349481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12624.388255] Lustre: server umount lustre-MDT0001 complete [12638.155125] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12641.893177] Key type lgssc unregistered [12642.388588] LNet: 353641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12642.407517] LNetError: 353641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12642.448905] LNet: Removed LNI 192.168.204.122@tcp [12643.728670] Key type .llcrypt unregistered [12643.732215] Key type ._llcrypt unregistered [12750.236398] Key type ._llcrypt registered [12750.239758] Key type .llcrypt registered [12750.339280] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12750.693543] Key type .llcrypt unregistered [12750.700409] Key type ._llcrypt unregistered [12825.698051] Key type ._llcrypt registered [12825.699979] Key type .llcrypt registered [12825.893337] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12826.224526] Key type .llcrypt unregistered [12826.228068] Key type ._llcrypt unregistered [12856.745479] Key type ._llcrypt registered [12856.751284] Key type .llcrypt registered [12856.956566] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12857.416515] Key type .llcrypt unregistered [12857.417933] Key type ._llcrypt unregistered [12888.904342] Key type ._llcrypt registered [12888.905815] Key type .llcrypt registered [12889.002630] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12889.336259] Key type .llcrypt unregistered [12889.339485] Key type ._llcrypt unregistered [12971.966992] Key type ._llcrypt registered [12971.969249] Key type .llcrypt registered [12972.128219] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12972.431802] Key type .llcrypt unregistered [12972.434259] Key type ._llcrypt unregistered [13008.519256] Key type ._llcrypt registered [13008.521277] Key type .llcrypt registered [13008.636967] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [13009.039685] Key type .llcrypt unregistered [13009.051577] Key type ._llcrypt unregistered [13040.307661] Key type ._llcrypt registered [13040.310531] Key type .llcrypt registered [13040.442952] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [13040.731421] Key type .llcrypt unregistered [13040.734317] Key type ._llcrypt unregistered [13069.519397] Key type ._llcrypt registered [13069.522612] Key type .llcrypt registered [13069.646500] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [13069.984324] Key type .llcrypt unregistered [13069.989287] Key type ._llcrypt unregistered [13096.846735] Key type ._llcrypt registered [13096.849355] Key type .llcrypt registered [13096.950772] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [13097.257392] Key type .llcrypt unregistered [13097.261886] Key type ._llcrypt unregistered [13128.364289] Key type ._llcrypt registered [13128.367178] Key type .llcrypt registered [13128.502883] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [13128.883299] Key type .llcrypt unregistered [13128.892690] Key type ._llcrypt unregistered [13191.241872] Key type ._llcrypt registered [13191.252447] Key type .llcrypt registered [13191.385525] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [13191.734283] Key type .llcrypt unregistered [13191.736042] Key type ._llcrypt unregistered [13224.786759] Key type ._llcrypt registered [13224.789052] Key type .llcrypt registered [13224.850241] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [13225.065460] Key type .llcrypt unregistered [13225.070875] Key type ._llcrypt unregistered [13255.590573] Key type ._llcrypt registered [13255.592842] Key type .llcrypt registered [13255.693571] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [13256.056512] Key type .llcrypt unregistered [13256.060361] Key type ._llcrypt unregistered [13311.012115] Key type ._llcrypt registered [13311.016742] Key type .llcrypt registered [13311.157383] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [13326.941991] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [13327.958105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13328.131521] alg: No test for adler32 (adler32-zlib) [13329.236671] Lustre: Lustre: Build Version: 2.17.57_133_g066c38c [13329.511543] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [13331.175401] Key type lgssc registered [13332.391650] Lustre: Echo OBD driver; http://www.lustre.org/ [13369.291107] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13369.319599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13370.668397] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13370.814924] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13370.954356] Lustre: tst250fs-MDT0000: new disk, initializing [13371.052333] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13371.095830] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13375.093251] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [13384.504154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13384.625327] Lustre: 374048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13384.684105] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13384.687255] Lustre: Skipped 1 previous similar message [13384.751235] Lustre: tst250fs-MDT0001: new disk, initializing [13384.885639] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13384.953189] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13384.960210] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13389.296544] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [13394.095702] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13400.386176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13400.837317] Lustre: tst250fs-OST0000: new disk, initializing [13400.854399] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13400.868478] Lustre: 375680:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13401.047498] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13405.792234] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13405.815732] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13405.934241] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13407.793080] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [13417.900757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13418.150615] Lustre: tst250fs-OST0001: new disk, initializing [13418.156267] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13418.162081] Lustre: 376562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13418.312322] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13423.747099] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13423.760214] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13423.850791] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13425.011546] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [13430.530321] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13432.912654] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13432.916605] Lustre: Skipped 2 previous similar messages [13451.749582] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13451.767841] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13451.796127] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13454.823255] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13454.827708] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13454.851294] Lustre: Skipped 1 previous similar message [13454.858162] Lustre: Skipped 2 previous similar messages [13456.707163] Lustre: server umount tst250fs-MDT0000 complete [13460.826296] LustreError: 375678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142521 with bad export cookie 9830608761467015088 [13460.837850] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13460.842885] LustreError: 375678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13461.342855] Lustre: server umount tst250fs-MDT0001 complete [13475.786355] Lustre: server umount tst250fs-OST0000 complete [13490.284356] Lustre: server umount tst250fs-OST0001 complete [13507.136162] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [13515.455957] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [13545.884456] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:03:24 (1789142604) [13547.346371] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13563.000067] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [13572.850716] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [13604.295425] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13284 sec ========= 12:04:22 (1789142662) [13605.972798] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:04:24 (1789142664) === [13610.244857] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:04:28 (1789142668) === [13650.153510] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [13653.135288] Key type lgssc unregistered [13653.420792] LNet: 385705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13653.434306] LNetError: 385705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13653.450190] LNet: Removed LNI 192.168.204.122@tcp [13654.262158] Key type .llcrypt unregistered [13654.265211] Key type ._llcrypt unregistered