[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492893935 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2656MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003224] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008007] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011145] LSM: Security Framework initializing [ 0.012068] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014081] *** VALIDATE selinux *** [ 0.022760] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027297] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028139] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031139] *** VALIDATE tmpfs *** [ 0.033523] *** VALIDATE proc *** [ 0.035011] *** VALIDATE cgroup *** [ 0.035866] *** VALIDATE cgroup2 *** [ 0.036268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039041] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042021] Spectre V2 : User space: Vulnerable [ 0.043009] Speculative Store Bypass: Vulnerable [ 0.045687] debug: unmapping init [mem 0xffffffff96259000-0xffffffff96260fff] [ 0.047183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048752] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049026] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056287] rcu: Hierarchical SRCU implementation. [ 0.058433] smp: Bringing up secondary CPUs ... [ 0.059666] x86: Booting SMP configuration: [ 0.060034] .... node #0, CPUs: #1 #2 #3 [ 0.063752] smp: Brought up 1 node, 4 CPUs [ 0.065016] smpboot: Max logical packages: 1 [ 0.066019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.166026] node 0 deferred pages initialised in 98ms [ 0.172173] devtmpfs: initialized [ 0.173253] x86/mm: Memory block size: 128MB [ 0.174840] gcov: version magic: 0x41383552 [ 0.181065] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.185094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.188320] pinctrl core: initialized pinctrl subsystem [ 0.190214] [ 0.190623] ************************************************************* [ 0.192016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195014] ** ** [ 0.197012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.200014] ** ** [ 0.202011] ** This means that this kernel is built to expose internal ** [ 0.204012] ** IOMMU data structures, which may compromise security on ** [ 0.206014] ** your system. ** [ 0.209015] ** ** [ 0.211013] ** If you see this message and you are not debugging the ** [ 0.214012] ** kernel, report this immediately to your vendor! ** [ 0.216011] ** ** [ 0.218013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.221013] ************************************************************* [ 0.223627] NET: Registered protocol family 16 [ 0.225449] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.228057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.230059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.233110] cpuidle: using governor menu [ 0.234681] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.237497] PCI: Using configuration type 1 for base access [ 0.239119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.250120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.252040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.255158] cryptd: max_cpu_qlen set to 1000 [ 0.258268] ACPI: Added _OSI(Module Device) [ 0.260018] ACPI: Added _OSI(Processor Device) [ 0.262015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.264013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.268978] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.276142] ACPI: Interpreter enabled [ 0.277065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.278015] ACPI: Using IOAPIC for interrupt routing [ 0.280109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.284405] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.295153] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.297037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.299017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.302064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.306482] acpiphp: Slot [2] registered [ 0.308146] acpiphp: Slot [5] registered [ 0.310128] acpiphp: Slot [6] registered [ 0.311125] acpiphp: Slot [7] registered [ 0.313167] acpiphp: Slot [8] registered [ 0.315106] acpiphp: Slot [9] registered [ 0.316140] acpiphp: Slot [10] registered [ 0.318142] acpiphp: Slot [3] registered [ 0.319104] acpiphp: Slot [4] registered [ 0.321110] acpiphp: Slot [11] registered [ 0.322121] acpiphp: Slot [12] registered [ 0.324105] acpiphp: Slot [13] registered [ 0.326119] acpiphp: Slot [14] registered [ 0.327146] acpiphp: Slot [15] registered [ 0.328102] acpiphp: Slot [16] registered [ 0.330102] acpiphp: Slot [17] registered [ 0.332109] acpiphp: Slot [18] registered [ 0.333158] acpiphp: Slot [19] registered [ 0.335123] acpiphp: Slot [20] registered [ 0.336140] acpiphp: Slot [21] registered [ 0.338106] acpiphp: Slot [22] registered [ 0.340121] acpiphp: Slot [23] registered [ 0.341101] acpiphp: Slot [24] registered [ 0.343095] acpiphp: Slot [25] registered [ 0.344107] acpiphp: Slot [26] registered [ 0.346139] acpiphp: Slot [27] registered [ 0.347132] acpiphp: Slot [28] registered [ 0.348114] acpiphp: Slot [29] registered [ 0.350098] acpiphp: Slot [30] registered [ 0.352087] acpiphp: Slot [31] registered [ 0.354096] PCI host bridge to bus 0000:00 [ 0.356027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.359012] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.361031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.364057] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.367029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.370015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.371152] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373049] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.377348] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.390016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.395061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.398020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.400024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.404022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.407486] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.410857] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.414044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.417810] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.423015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.437031] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.443015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.450000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.465020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.472018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.493021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.503007] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.508011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.513017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.538000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.544015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.546013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.555012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.563704] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.572019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.581016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.618021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.631055] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.638018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.645014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.666017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.678279] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.686020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.694016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.709020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.725129] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.728398] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.730380] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.734411] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.737314] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.743151] iommu: Default domain type: Passthrough [ 0.745445] SCSI subsystem initialized [ 0.747132] ACPI: bus type USB registered [ 0.749141] usbcore: registered new interface driver usbfs [ 0.752100] usbcore: registered new interface driver hub [ 0.755094] usbcore: registered new device driver usb [ 0.757187] pps_core: LinuxPPS API ver. 1 registered [ 0.760017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.764084] PTP clock support registered [ 0.767067] EDAC MC: Ver: 3.0.0 [ 0.769510] PCI: Using ACPI for IRQ routing [ 0.771770] NetLabel: Initializing [ 0.773010] NetLabel: domain hash size = 128 [ 0.775011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.777082] NetLabel: unlabeled traffic allowed by default [ 0.779117] vgaarb: loaded [ 0.780249] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.783017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.788338] clocksource: Switched to clocksource kvm-clock [ 0.895583] VFS: Disk quotas dquot_6.6.0 [ 0.897821] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.901528] *** VALIDATE ramfs *** [ 0.903425] *** VALIDATE hugetlbfs *** [ 0.905564] pnp: PnP ACPI init [ 0.908510] pnp: PnP ACPI: found 6 devices [ 0.928150] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.932160] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.934513] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.936850] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.938899] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.941106] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.943818] NET: Registered protocol family 2 [ 0.946302] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.951144] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.954785] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.959931] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.963302] TCP: Hash tables configured (established 65536 bind 65536) [ 0.966145] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.969124] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971893] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.974675] NET: Registered protocol family 1 [ 0.977043] RPC: Registered named UNIX socket transport module. [ 0.979269] RPC: Registered udp transport module. [ 0.981217] RPC: Registered tcp transport module. [ 0.983123] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985707] NET: Registered protocol family 44 [ 0.987414] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.989298] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.991442] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.993473] PCI: CLS 0 bytes, default 64 [ 0.995218] Unpacking initramfs... [ 2.310669] debug: unmapping init [mem 0xffff8b317cc54000-0xffff8b317ffbffff] [ 2.313969] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.316067] software IO TLB: mapped [mem 0x00000000a2000000-0x00000000a6000000] (64MB) [ 2.318205] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.817903] Initialise system trusted keyrings [ 2.819859] Key type blacklist registered [ 2.821741] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.831374] zbud: loaded [ 2.834525] *** VALIDATE nfs *** [ 2.835769] *** VALIDATE nfs4 *** [ 2.837066] pstore: using deflate compression [ 2.840714] Platform Keyring initialized [ 2.956836] NET: Registered protocol family 38 [ 2.959017] Key type asymmetric registered [ 2.960800] Asymmetric key parser 'x509' registered [ 2.962575] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.965793] io scheduler mq-deadline registered [ 2.967672] io scheduler kyber registered [ 2.969488] io scheduler bfq registered [ 2.971464] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.974818] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.977839] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.980759] ACPI: Power Button [PWRF] [ 2.986553] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.994440] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.010517] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.018365] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.033320] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.061581] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.090272] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.095435] Non-volatile memory driver v1.3 [ 3.097309] Linux agpgart interface v0.103 [ 3.129206] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.132220] vda: detected capacity change from 0 to 74702848 [ 3.174358] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.177228] vdb: detected capacity change from 0 to 1073741824 [ 3.198533] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.201450] vdc: detected capacity change from 0 to 2621440000 [ 3.220879] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.223462] vdd: detected capacity change from 0 to 2621440000 [ 3.241347] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.244502] vde: detected capacity change from 0 to 4294967296 [ 3.265502] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.268734] vdf: detected capacity change from 0 to 4294967296 [ 3.281287] libphy: Fixed MDIO Bus: probed [ 3.287676] usbcore: registered new interface driver usbserial_generic [ 3.290724] usbserial: USB Serial support registered for generic [ 3.292947] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.297842] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.300069] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.302692] mousedev: PS/2 mouse device common for all mice [ 3.305486] rtc_cmos 00:05: RTC can wake from S4 [ 3.308180] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.312161] rtc_cmos 00:05: registered as rtc0 [ 3.315829] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.316053] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.319855] intel_pstate: CPU model not supported [ 3.325634] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.329693] hid: raw HID events driver (C) Jiri Kosina [ 3.332019] usbcore: registered new interface driver usbhid [ 3.334189] usbhid: USB HID core driver [ 3.335634] drop_monitor: Initializing network drop monitor service [ 3.338206] Initializing XFRM netlink socket [ 3.340493] NET: Registered protocol family 10 [ 3.343759] Segment Routing with IPv6 [ 3.345139] NET: Registered protocol family 17 [ 3.347270] mpls_gso: MPLS GSO support [ 3.353130] RAS: Correctable Errors collector initialized. [ 3.355489] AVX version of gcm_enc/dec engaged. [ 3.357170] AES CTR mode by8 optimization enabled [ 3.442739] sched_clock: Marking stable (3442595913, 0)->(4376243713, -933647800) [ 3.446498] registered taskstats version 1 [ 3.447888] Loading compiled-in X.509 certificates [ 3.449446] zswap: loaded using pool lzo/zbud [ 3.471767] Key type big_key registered [ 3.481313] Key type encrypted registered [ 3.482487] ima: No TPM chip found, activating TPM-bypass! [ 3.483812] ima: Allocated hash algorithm: sha1 [ 3.484919] ima: No architecture policies found [ 3.486117] evm: Initialising EVM extended attributes: [ 3.487365] evm: security.selinux [ 3.488117] evm: security.ima [ 3.488766] evm: security.capability [ 3.489697] evm: HMAC attrs: 0x1 [ 3.491547] rtc_cmos 00:05: setting system clock to 2026-08-14 02:18:27 UTC (1786673907) [ 3.496726] debug: unmapping init [mem 0xffffffff97203000-0xffffffff973fffff] [ 3.498832] debug: unmapping init [mem 0xffffffff95f82000-0xffffffff96258fff] [ 3.507112] Write protecting the kernel read-only data: 28672k [ 3.509684] debug: unmapping init [mem 0xffffffff94603000-0xffffffff947fffff] [ 3.511655] debug: unmapping init [mem 0xffffffff94f14000-0xffffffff94ffffff] [ 3.538403] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.543746] systemd[1]: Detected virtualization kvm. [ 3.546076] systemd[1]: Detected architecture x86-64. [ 3.548504] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.578156] systemd[1]: No hostname configured. [ 3.580270] systemd[1]: Set hostname to . [ 3.582913] random: systemd: uninitialized urandom read (16 bytes read) [ 3.585795] systemd[1]: Initializing machine ID from random generator. [ 3.645523] random: ln: uninitialized urandom read (6 bytes read) [ 3.734687] random: systemd: uninitialized urandom read (16 bytes read) [ 3.738353] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.743558] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.747990] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.409791] device-mapper: uevent: version 1.0.3 [ 4.412332] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.273066] random: fast init done [ 5.301995] virtio_net virtio0 ens2: renamed from eth0 [ 5.381377] scsi host0: ata_piix [ 5.400807] scsi host1: ata_piix [ 5.402535] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.405093] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.771593] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.994289] random: crng init done [ 9.995456] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.481529] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.633442] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.899962] SELinux: Disabled at runtime. [ 11.963493] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.972319] systemd[1]: Detected virtualization kvm. [ 11.974303] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.503728] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.507458] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.512123] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.516301] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.519805] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.530830] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.538064] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 12.572738] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.065819] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.438651] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.486641] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.637162] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.648406] EDAC sbridge: Ver: 1.1.2 [ 15.292154] Key type dns_resolver registered [ 15.598775] NFS: Registering the id_resolver key type [ 15.600861] Key type id_resolver registered [ 15.602402] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg215-server login: [ 42.004944] libcfs: loading out-of-tree module taints kernel. [ 42.029854] Key type ._llcrypt registered [ 42.031770] Key type .llcrypt registered [ 42.089130] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 52.512616] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 54.357630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.366557] alg: No test for adler32 (adler32-zlib) [ 55.783059] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 56.806750] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 57.023007] hrtimer: interrupt took 8972844 ns [ 58.671160] Key type lgssc registered [ 60.077465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.190400] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 128.665530] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 146.802207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.853173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.181552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 148.201260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 148.301890] Lustre: lustre-MDT0000: new disk, initializing [ 148.420139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.453749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.167388] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.664637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.782393] Lustre: 6489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.820862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.828326] Lustre: Skipped 1 previous similar message [ 167.968360] Lustre: lustre-MDT0001: new disk, initializing [ 168.126453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 168.179243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 168.196359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 172.945671] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.416472] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.621575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 189.828937] Lustre: lustre-OST0000: new disk, initializing [ 189.832867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 189.836883] Lustre: 8428:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 189.880528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.926490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.944417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.080591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 196.851961] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.407080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.544842] Lustre: lustre-OST0001: new disk, initializing [ 215.548314] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 215.553863] Lustre: 9502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.623812] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.317126] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.295846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 222.316714] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 222.391670] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 234.349931] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.550815] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 253.206404] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing check_logdir /tmp/testlogs/ [ 259.314581] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing yml_node [ 267.973612] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 271.811952] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 274.500859] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 276.700421] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 22:22:59 EDT 2026 [ 295.664251] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 297.226279] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 314.851289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.855878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.876530] Lustre: Skipped 1 previous similar message [ 314.899673] Lustre: Skipped 3 previous similar messages [ 319.987724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.997338] Lustre: Skipped 2 previous similar messages [ 320.907571] Lustre: server umount lustre-MDT0000 complete [ 324.165615] LustreError: 8427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786674228 with bad export cookie 13589034744502010331 [ 324.178692] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.186933] LustreError: 8427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 325.089795] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.100474] Lustre: Skipped 2 previous similar messages [ 325.111046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.114934] Lustre: Skipped 2 previous similar messages [ 329.127310] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.144429] Lustre: Skipped 1 previous similar message [ 330.587831] Lustre: server umount lustre-MDT0001 complete [ 334.885848] Lustre: server umount lustre-OST0000 complete [ 339.020962] Lustre: server umount lustre-OST0001 complete [ 348.869781] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 360.707693] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 404.466704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.831571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 404.875597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 405.048517] Lustre: lustre-MDT0000: new disk, initializing [ 405.171235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.226606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 410.030802] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 422.830609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.008447] Lustre: 15056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 423.081613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 423.085748] Lustre: Skipped 1 previous similar message [ 423.163714] Lustre: lustre-MDT0001: new disk, initializing [ 423.276230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.337276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 423.359894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 428.574867] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 439.257532] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 446.203106] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 453.486498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 453.800413] Lustre: lustre-OST0000: new disk, initializing [ 453.805306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 453.814517] Lustre: 17025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 453.925797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.025569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 455.032978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 455.194263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 461.023239] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 470.621060] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 475.862299] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 476.035627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 481.494286] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 481.755748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 485.859734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.868163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.875550] LustreError: Skipped 1 previous similar message [ 485.879311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.905240] Lustre: Skipped 2 previous similar messages [ 489.894738] Lustre: server umount lustre-OST0000 complete [ 495.072671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 495.084127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.103465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.106935] Lustre: Skipped 1 previous similar message [ 499.442717] Lustre: server umount lustre-MDT0000 complete [ 501.220331] LustreError: 15068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.247442] LustreError: 15068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 503.357685] LustreError: 15050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786674407 with bad export cookie 13589034744502011864 [ 503.359096] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 503.374962] LustreError: 15050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 503.748321] Lustre: server umount lustre-MDT0001 complete [ 509.703984] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:26:52 (1786674412) [ 528.355445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.554911] LustreError: 19899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.633640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 533.065781] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 533.994721] LustreError: 19900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.084327] LustreError: 19899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.581477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.676757] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 556.138443] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 563.557931] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.286508] Lustre: Failing over lustre-MDT0000 [ 566.887508] Lustre: server umount lustre-MDT0000 complete [ 567.334071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.339283] Lustre: Skipped 1 previous similar message [ 567.347419] LustreError: 19899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.359710] LustreError: 19899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 577.505677] LustreError: 20627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.526197] LustreError: 20627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 583.648373] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786674471/real 1786674471] req@ffff8b30c35c2300 x1873463434448128/t0(0) o400->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786674487 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 583.675858] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.569563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.888598] LustreError: 20627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.921383] LustreError: 20627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 594.914823] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b31d6303480 x1873463434451328/t0(0) o250->MGC192.168.202.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 595.316645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 595.323878] Lustre: Skipped 1 previous similar message [ 600.233387] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 600.549461] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 600.555799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 600.570487] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 610.552640] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 614.786572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.805708] Lustre: Skipped 1 previous similar message [ 614.816938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 614.823118] Lustre: Skipped 2 previous similar messages [ 629.215199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 629.607982] Lustre: server umount lustre-MDT0000 complete [ 633.093308] LustreError: 19879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786674537 with bad export cookie 13589034744502013278 [ 633.095720] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.105570] LustreError: 19879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 633.473718] Lustre: server umount lustre-MDT0001 complete [ 641.999322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.420959] LustreError: 23237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.442168] LustreError: 23237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 646.318986] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 654.672133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.856298] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 665.514768] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 670.863173] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 673.040066] Lustre: Failing over lustre-MDT0000 [ 673.301848] Lustre: server umount lustre-MDT0000 complete [ 675.303623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 675.314173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.341445] Lustre: Skipped 1 previous similar message [ 692.275919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 692.485519] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 692.833813] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 697.712605] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 697.836462] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 697.851645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 697.870629] Lustre: Skipped 1 previous similar message [ 697.889675] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 707.139722] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 713.190438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.200546] Lustre: Skipped 1 previous similar message [ 713.209031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 713.214455] Lustre: Skipped 8 previous similar messages [ 717.609642] Lustre: server umount lustre-MDT0000 complete [ 718.307980] LustreError: 23238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.332281] LustreError: 23238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 722.033749] LustreError: 23219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786674626 with bad export cookie 13589034744502014216 [ 722.065862] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.393307] Lustre: server umount lustre-MDT0001 complete [ 731.207451] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:30:34 (1786674634) [ 747.189329] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 757.275475] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 800.762841] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 810.677098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.861208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 810.899417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 811.032609] Lustre: lustre-MDT0000: new disk, initializing [ 811.118868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.124543] Lustre: Skipped 3 previous similar messages [ 811.140248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 815.985450] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 826.583084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.675360] Lustre: 29611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 826.965144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 832.105355] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 842.020335] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 848.472954] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 856.011586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 856.270370] Lustre: 31584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 857.905526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 857.965926] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 858.008745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 858.030480] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 862.828763] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 873.806818] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 879.259369] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 899.047560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 899.059553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.070492] Lustre: Skipped 1 previous similar message [ 899.082046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.088923] Lustre: Skipped 1 previous similar message [ 903.687457] Lustre: server umount lustre-OST0000 complete [ 908.768211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 908.774662] LustreError: Skipped 1 previous similar message [ 913.849543] Lustre: server umount lustre-MDT0000 complete [ 914.406146] LustreError: 29621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.429675] LustreError: 29621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 917.721924] LustreError: 31583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786674821 with bad export cookie 13589034744502014622 [ 917.724144] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 917.730948] LustreError: 31583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 918.430536] Lustre: server umount lustre-MDT0001 complete [ 932.676168] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 935.661216] Key type lgssc unregistered [ 936.080754] LNet: 33564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.101022] LNetError: 33564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.133351] LNet: Removed LNI 192.168.202.115@tcp [ 938.105296] Key type .llcrypt unregistered [ 938.107120] Key type ._llcrypt unregistered [ 965.136474] Key type ._llcrypt registered [ 965.138250] Key type .llcrypt registered [ 965.269234] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 966.608635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.744533] alg: No test for adler32 (adler32-zlib) [ 967.819733] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 968.055804] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 969.783227] Key type lgssc registered [ 971.309389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 979.642426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 979.686935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.095940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.806739] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 991.716798] LustreError: 34978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 991.731816] LustreError: 34978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 994.208394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.667228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 999.775348] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1007.620835] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1012.662947] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1018.941572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1019.236980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.285940] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1024.767160] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1026.433493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1031.789882] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1045.899521] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:35:49 (1786674949) [ 1047.584342] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1049.663299] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1057.234487] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:35:59 (1786674959) [ 1066.791025] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:36:09 (1786674969) [ 1068.080632] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1076.901206] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1076.908410] Lustre: Skipped 2 previous similar messages [ 1078.878690] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:36:21 (1786674981) [ 1080.946555] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1080.949322] Lustre: Skipped 2 previous similar messages [ 1099.377239] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1099.383222] Lustre: Skipped 7 previous similar messages [ 1102.210451] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:36:45 (1786675005) [ 1104.980330] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1108.426542] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1117.613614] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:37:00 (1786675020) [ 1119.650396] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1119.658798] Lustre: Skipped 3 previous similar messages [ 1132.826273] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:37:15 (1786675035) [ 1148.385071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1148.400097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.419980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1149.415150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.416051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1149.457671] Lustre: Skipped 1 previous similar message [ 1153.882538] Lustre: server umount lustre-OST0000 complete [ 1158.634788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1158.647830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.677483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.656224] Lustre: server umount lustre-MDT0000 complete [ 1164.774586] LustreError: 34979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.798896] LustreError: 34979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1169.147811] LustreError: 39384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786675073 with bad export cookie 14435071274862324834 [ 1169.151072] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.160939] LustreError: 39384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1169.637054] Lustre: server umount lustre-MDT0001 complete [ 1181.094678] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1184.245841] Key type lgssc unregistered [ 1184.528686] LNet: 42833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.535778] LNetError: 42833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.563809] LNet: Removed LNI 192.168.202.115@tcp [ 1185.489901] Key type .llcrypt unregistered [ 1185.494087] Key type ._llcrypt unregistered [ 1203.864889] Key type ._llcrypt registered [ 1203.868543] Key type .llcrypt registered [ 1203.990530] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 1222.400957] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1223.306062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1223.403891] alg: No test for adler32 (adler32-zlib) [ 1224.708313] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1225.058442] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1226.823512] Key type lgssc registered [ 1228.256416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1262.217281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1262.226628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.405723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1263.438822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1263.535530] Lustre: lustre-MDT0000: new disk, initializing [ 1263.603357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.624247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1267.685694] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1277.964130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1278.164914] Lustre: 45986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1278.256719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1278.261464] Lustre: Skipped 1 previous similar message [ 1278.399691] Lustre: lustre-MDT0001: new disk, initializing [ 1278.430791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1278.570640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1278.659634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1278.692623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1283.987617] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1293.993798] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1299.723671] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1307.820201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1308.055712] Lustre: lustre-OST0000: new disk, initializing [ 1308.061741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1308.077308] Lustre: 47959:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1308.191661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.916639] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1318.005594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1318.017146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1318.161121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1325.455026] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1329.608657] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1329.856956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1334.700780] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1334.916206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1338.851581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.860644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1338.879157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1338.896710] Lustre: Skipped 1 previous similar message [ 1343.213795] Lustre: server umount lustre-OST0000 complete [ 1349.089192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.092982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1349.116314] Lustre: Skipped 2 previous similar messages [ 1354.211995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1354.221331] Lustre: Skipped 2 previous similar messages [ 1358.307026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1358.316872] Lustre: Skipped 1 previous similar message [ 1361.375735] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1361.599920] Lustre: server umount lustre-MDT0000 complete [ 1365.239958] LustreError: 47958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786675269 with bad export cookie 8677673171137764368 [ 1365.250172] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1365.259115] LustreError: 47958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1365.505674] Lustre: server umount lustre-MDT0001 complete [ 1374.538336] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:41:17 (1786675277) [ 1380.281219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.872670] LustreError: 50421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.892480] LustreError: 50421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1380.969381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.586119] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1386.468984] LustreError: 50422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.562781] LustreError: 50421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.077407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.433577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1397.210801] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1404.430243] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1410.755882] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1418.036183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1418.149791] LustreError: 52240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1418.479477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.061500] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1432.872197] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1437.550803] LustreError: 52948:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1438.594977] Lustre: Modifying parameter general.timeout=1 in log params [ 1439.601426] Lustre: Modifying parameter general.timeout=2 in log params [ 1442.417786] Lustre: Modifying parameter general.timeout=5 in log params [ 1442.427189] Lustre: Skipped 2 previous similar messages [ 1446.671184] Lustre: Modifying parameter general.timeout=9 in log params [ 1446.677657] Lustre: Skipped 3 previous similar messages [ 1455.418934] Lustre: Modifying parameter general.timeout=18 in log params [ 1455.424820] Lustre: Skipped 8 previous similar messages [ 1471.585186] Lustre: Modifying parameter general.timeout=36 in log params [ 1471.588699] Lustre: Skipped 17 previous similar messages [ 1503.948636] Lustre: Modifying parameter general.timeout=76 in log params [ 1503.954000] Lustre: Skipped 39 previous similar messages [ 1559.574800] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:44:22 (1786675462) [ 1561.907405] Lustre: 59884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1563.696390] Lustre: 59982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1567.274597] Lustre: 60194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1567.284466] Lustre: 60194:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1568.285884] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1568.291579] Lustre: Skipped 59 previous similar messages [ 1598.112632] Lustre: 61746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1598.127867] Lustre: 61746:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1598.169656] LustreError: 61746:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1601.675659] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:45:04 (1786675504) [ 1613.794107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1613.804837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.827163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1616.364720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.879074] Lustre: server umount lustre-MDT0000 complete [ 1621.487056] LustreError: 50421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.515964] LustreError: 50421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1623.780583] LustreError: 62091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786675527 with bad export cookie 8677673171137765250 [ 1623.783082] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1623.788785] LustreError: 62091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1624.288631] Lustre: server umount lustre-MDT0001 complete [ 1637.572183] Lustre: server umount lustre-OST0000 complete [ 1663.027343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.373886] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1663.540971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.539948] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1680.150596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1680.229183] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1680.246972] Lustre: 64283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1684.627931] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1691.699059] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1697.339209] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1704.636291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1704.908185] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1705.325101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1705.331203] Lustre: Skipped 1 previous similar message [ 1713.078258] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1721.006712] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1722.796350] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1722.801982] Lustre: Skipped 17 previous similar messages [ 1722.840569] Lustre: 66849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1733.088414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1733.098098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.110910] Lustre: Skipped 1 previous similar message [ 1733.121150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1733.127619] Lustre: Skipped 3 previous similar messages [ 1737.409981] Lustre: server umount lustre-OST0000 complete [ 1741.795885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1741.813600] LustreError: Skipped 1 previous similar message [ 1741.816814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.824741] Lustre: Skipped 1 previous similar message [ 1748.380496] Lustre: server umount lustre-MDT0000 complete [ 1748.965606] LustreError: 65415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.996462] LustreError: 65415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1752.821097] LustreError: 64276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786675656 with bad export cookie 8677673171137768218 [ 1752.823535] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1752.832573] LustreError: 64276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1753.243393] Lustre: server umount lustre-MDT0001 complete [ 1768.566748] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1771.346168] Key type lgssc unregistered [ 1771.709370] LNet: 68063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.724637] LNetError: 68063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.755898] LNet: Removed LNI 192.168.202.115@tcp [ 1772.493544] Key type .llcrypt unregistered [ 1772.495051] Key type ._llcrypt unregistered [ 1801.284628] Key type ._llcrypt registered [ 1801.287965] Key type .llcrypt registered [ 1801.444097] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1803.084827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1803.338821] alg: No test for adler32 (adler32-zlib) [ 1804.376740] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1804.572424] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1806.239264] Key type lgssc registered [ 1807.375546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1814.363467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1814.380133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.070437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.541397] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1825.251496] LustreError: 69478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1825.278230] LustreError: 69478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1830.377723] LustreError: 69479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.877207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.660655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.739465] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1845.521084] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1851.993894] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1859.963315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1860.406759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.088992] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1874.793765] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1883.276957] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1892.845876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1892.854096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.864288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.441762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.461303] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.346301] Lustre: server umount lustre-MDT0000 complete [ 1902.560796] LustreError: 69478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1902.580829] LustreError: 69478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1903.163130] LustreError: 69458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786675807 with bad export cookie 9195873842849064935 [ 1903.168237] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1903.174117] LustreError: 69458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1903.536738] Lustre: server umount lustre-MDT0001 complete [ 1918.012430] Lustre: server umount lustre-OST0000 complete [ 1948.876889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.248719] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1949.304766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1949.731629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.152572] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1963.668480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.835236] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1963.883564] Lustre: 74348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1964.648302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.791242] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1977.572237] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1984.684307] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1991.915877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1992.109270] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1992.622618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.033905] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2008.090116] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2009.759898] Lustre: 76929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2012.178229] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2021.856989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2021.865962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.887517] Lustre: Skipped 1 previous similar message [ 2021.897432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2021.906617] Lustre: Skipped 2 previous similar messages [ 2024.128781] Lustre: server umount lustre-OST0000 complete [ 2031.074080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2031.077526] LustreError: Skipped 1 previous similar message [ 2031.080608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.097203] Lustre: Skipped 1 previous similar message [ 2031.101499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.111818] Lustre: Skipped 1 previous similar message [ 2035.015307] Lustre: server umount lustre-MDT0000 complete [ 2037.219606] LustreError: 75384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.248958] LustreError: 75384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2039.875434] LustreError: 74341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786675943 with bad export cookie 9195873842849065775 [ 2039.881915] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.885785] LustreError: 74341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2040.355485] Lustre: server umount lustre-MDT0001 complete [ 2054.228229] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 2057.533980] Key type lgssc unregistered [ 2057.947869] LNet: 78141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.965016] LNetError: 78141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.988119] LNet: Removed LNI 192.168.202.115@tcp [ 2060.236614] Key type .llcrypt unregistered [ 2060.241093] Key type ._llcrypt unregistered [ 2087.204818] Key type ._llcrypt registered [ 2087.207408] Key type .llcrypt registered [ 2087.330831] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2088.980518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2089.058297] alg: No test for adler32 (adler32-zlib) [ 2090.216991] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2090.610759] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 2092.335825] Key type lgssc registered [ 2093.501382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2102.872865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2102.892278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.551202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.156183] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2114.020394] LustreError: 79558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.033123] LustreError: 79558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2119.143684] LustreError: 79557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.081897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.829128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2126.189889] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2133.832520] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2139.673663] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2148.655236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2149.425680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.824242] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2166.027707] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2173.366733] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2174.366649] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2176.446275] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2176.454062] Lustre: Skipped 1 previous similar message [ 2179.136416] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2179.139217] Lustre: Skipped 2 previous similar messages [ 2184.122786] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2184.130934] Lustre: Skipped 4 previous similar messages [ 2192.955973] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2192.967891] Lustre: Skipped 8 previous similar messages [ 2209.235739] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2209.240231] Lustre: Skipped 17 previous similar messages [ 2242.022009] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2242.028517] Lustre: Skipped 35 previous similar messages [ 2272.663988] Lustre: 87653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2275.911688] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:56:18 (1786676178) [ 2277.800811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2279.356388] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:56:22 (1786676182) [ 2283.489349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2283.490247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.494530] LustreError: Skipped 1 previous similar message [ 2283.501723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2283.506188] Lustre: Skipped 1 previous similar message [ 2288.610610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2288.622658] Lustre: Skipped 1 previous similar message [ 2289.632876] Lustre: server umount lustre-OST0000 complete [ 2294.756154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2294.768072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.782941] Lustre: Skipped 1 previous similar message [ 2294.785909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2298.848412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2298.848973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.969274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2303.981191] Lustre: Skipped 2 previous similar messages [ 2308.063255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2308.288753] Lustre: server umount lustre-MDT0000 complete [ 2309.090758] LustreError: 79558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.100727] LustreError: 79558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2311.674464] LustreError: 79537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786676215 with bad export cookie 17586430967631514178 [ 2311.684744] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2311.688459] LustreError: 79537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2312.051194] Lustre: server umount lustre-MDT0001 complete [ 2324.665294] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 2327.934982] Key type lgssc unregistered [ 2328.187507] LNet: 88905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.198620] LNetError: 88905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.235644] LNet: Removed LNI 192.168.202.115@tcp [ 2329.117372] Key type .llcrypt unregistered [ 2329.122272] Key type ._llcrypt unregistered [ 2333.504260] Key type ._llcrypt registered [ 2333.506133] Key type .llcrypt registered [ 2333.599522] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2340.408370] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules [ 2342.339250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.402441] alg: No test for adler32 (adler32-zlib) [ 2343.730829] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2343.929372] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 2345.519552] Key type lgssc registered [ 2347.197865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2349.072381] LustreError: 90173:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2366.708212] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2370.467485] LustreError: 90173:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2376.988538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2377.030568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.738537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.956339] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2388.961019] LustreError: 91181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.979381] LustreError: 91181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2393.876531] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:58:16 (1786676296) [ 2394.086259] LustreError: 91182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.219169] LustreError: 91181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.280726] LustreError: 91214:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2399.292775] LustreError: 91214:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2399.301558] LustreError: 91214:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2399.796600] Lustre: server umount lustre-MDT0000 complete [ 2415.389876] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 2418.554766] Key type lgssc unregistered [ 2418.834651] LNet: 92710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2418.840056] LNetError: 92710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.886744] LNet: Removed LNI 192.168.202.115@tcp [ 2420.891540] Key type .llcrypt unregistered [ 2420.896333] Key type ._llcrypt unregistered [ 2441.789722] Key type ._llcrypt registered [ 2441.791949] Key type .llcrypt registered [ 2441.959802] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2444.183155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2444.216412] alg: No test for adler32 (adler32-zlib) [ 2445.522778] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2445.842057] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 2447.551757] Key type lgssc registered [ 2448.694565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2457.046619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2457.095250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.891048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.224963] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2468.327337] LustreError: 93929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.358696] LustreError: 93929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2471.018906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.817617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2476.920427] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2484.642907] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2490.217793] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2497.675771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.787385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.060564] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2515.090536] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2563.662396] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:01:06 (1786676466) [ 2565.787696] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2567.759383] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:01:10 (1786676470) [ 2572.772791] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.777038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.803187] Lustre: Skipped 1 previous similar message [ 2572.823719] Lustre: Skipped 2 previous similar messages [ 2577.907177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.920864] Lustre: Skipped 2 previous similar messages [ 2578.195506] Lustre: server umount lustre-MDT0000 complete [ 2582.028229] LustreError: 93909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786676486 with bad export cookie 4770857400584117309 [ 2582.029694] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.036899] LustreError: 93909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2582.845770] Lustre: server umount lustre-MDT0001 complete [ 2597.365207] Lustre: server umount lustre-OST0000 complete [ 2610.689226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.110794] LustreError: 98214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.302856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.290509] LustreError: 98215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.606826] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2621.421213] LustreError: 98214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.529568] LustreError: 98215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.982149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.640212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2632.068670] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2639.967967] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2645.947968] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2662.238240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2662.470671] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2662.476253] LustreError: 98205:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2662.495872] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2662.502851] LustreError: 100298:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2662.507628] LustreError: 100298:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2662.512411] LustreError: 100298:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2662.587629] Lustre: server umount lustre-OST0000 complete [ 2662.593516] LustreError: 100298:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2671.954572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2672.169965] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2672.173522] LustreError: 98203:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2672.196135] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2672.205600] LustreError: 100762:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2672.210388] LustreError: 100762:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2672.217344] LustreError: 100762:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2672.277155] Lustre: server umount lustre-OST0000 complete [ 2672.281350] LustreError: 100762:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2683.370028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.479526] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2683.481658] Lustre: Found index 0 for lustre-OST0000, updating log [ 2683.484205] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2683.512241] Lustre: lustre-OST0000: new disk, initializing [ 2683.513887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2683.516500] Lustre: 101299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2683.744016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.284857] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2695.153733] Lustre: Failing over lustre-OST0000 [ 2695.372883] Lustre: server umount lustre-OST0000 complete [ 2702.304774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.316478] Lustre: Skipped 1 previous similar message [ 2702.321510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2705.948654] Lustre: server umount lustre-MDT0000 complete [ 2707.424637] LustreError: 98214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.457033] LustreError: 98214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2709.749761] LustreError: 98196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786676613 with bad export cookie 4770857400584120046 [ 2709.751821] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2709.766968] LustreError: 98196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2710.030626] Lustre: server umount lustre-MDT0001 complete [ 2718.522837] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:03:41 (1786676621) [ 2733.809859] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2743.722896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.109137] LustreError: 103645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.273708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.514281] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2757.720505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.805398] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2765.348293] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2771.728687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.055778] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2788.020668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.079678] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2788.150469] Lustre: lustre-OST0001: new disk, initializing [ 2788.164389] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2788.181131] Lustre: 105964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2788.426807] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2788.429255] Lustre: Skipped 2 previous similar messages [ 2794.908106] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2797.206454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2797.218761] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2797.344530] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2803.491765] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2807.625811] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2812.902471] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.906195] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2812.920086] Lustre: Skipped 2 previous similar messages [ 2812.943133] Lustre: Skipped 3 previous similar messages [ 2817.789192] Lustre: server umount lustre-MDT0001 complete [ 2818.020364] LustreError: 103645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.043769] LustreError: 103645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2826.188692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.255996] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2826.259420] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2826.306520] Lustre: 103634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2827.739447] LustreError: 107514:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2827.749933] LustreError: 107514:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2827.762897] LustreError: 103637:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2827.868520] LustreError: 107525:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.115@tcp: cfg command failed: rc = -17 [ 2827.885415] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2827.885415] [ 2831.347511] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2832.376173] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2841.012260] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:05:43 (1786676743) [ 2859.170338] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2877.780122] Lustre: 109579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2899.359958] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2903.574711] Lustre: 110715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2985.441514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2985.457399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.471626] Lustre: Skipped 1 previous similar message [ 2985.479706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2985.487497] Lustre: Skipped 1 previous similar message [ 2991.623230] Lustre: server umount lustre-MDT0000 complete [ 2995.132500] LustreError: 106684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786676899 with bad export cookie 4770857400584121047 [ 2995.137629] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2995.142367] LustreError: 106684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2995.401909] Lustre: server umount lustre-MDT0001 complete [ 3009.468952] Lustre: server umount lustre-OST0000 complete [ 3012.575883] Lustre: 93242:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786676900/real 1786676900] req@ffff8b30c3150e00 x1873465939679872/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786676916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3012.624695] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.647657] Lustre: Skipped 3 previous similar messages [ 3013.987924] Lustre: server umount lustre-OST0001 complete [ 3019.530538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3029.712469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3040.938117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3052.369431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3070.023781] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3079.958737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.006193] Lustre: lustre-MDT0000: reset Object Index mappings [ 3080.421457] LustreError: 114662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.433515] LustreError: 114662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3080.522692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.530039] Lustre: Skipped 1 previous similar message [ 3085.681554] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3094.289312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.308935] Lustre: lustre-MDT0001: reset Object Index mappings [ 3099.954797] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3103.282780] Lustre: 115811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3109.276649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.945349] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3119.314508] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3124.319270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3125.831820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3125.839869] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3125.841647] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3131.516244] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3139.293990] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3142.611268] Lustre: 117689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3186.928716] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:11:29 (1786677089) [ 3217.889994] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.901334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3217.913033] Lustre: Skipped 3 previous similar messages [ 3217.922020] Lustre: Skipped 10 previous similar messages [ 3220.000089] Lustre: server umount lustre-MDT0000 complete [ 3223.017123] LustreError: 115399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.037276] LustreError: 115399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3223.962434] LustreError: 114643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786677127 with bad export cookie 4770857400584167912 [ 3223.967663] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3223.980463] LustreError: 114643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3234.283273] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3234.292369] Lustre: Skipped 5 previous similar messages [ 3238.367553] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3238.796079] Lustre: server umount lustre-MDT0001 complete [ 3248.901731] Lustre: server umount lustre-OST0000 complete [ 3252.527737] LustreError: 118652:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0x6d:0x0].0x0 (ffff8b31e6566700) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3258.845601] Lustre: server umount lustre-OST0001 complete [ 3265.897705] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 3275.803761] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3315.410925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.672194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3315.696394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3315.766507] Lustre: lustre-MDT0000: new disk, initializing [ 3315.822693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.830932] Lustre: Skipped 3 previous similar messages [ 3315.841577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3320.716334] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3327.666383] Lustre: server umount lustre-MDT0000 complete [ 3337.486904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.691057] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3337.699580] Lustre: 121962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3337.711251] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3345.583381] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3353.178228] Lustre: server umount lustre-MDT0000 complete [ 3361.800250] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:14:25 (1786677265) [ 3363.259108] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3365.209044] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:14:28 (1786677268) [ 3384.909843] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3394.700310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.678926] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3407.404090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.523460] Lustre: 124016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3407.564467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3407.567644] Lustre: Skipped 1 previous similar message [ 3407.616382] Lustre: lustre-MDT0001: new disk, initializing [ 3407.720812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3407.733694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3411.791818] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3416.491559] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3422.456157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3422.701425] Lustre: lustre-OST0000: new disk, initializing [ 3422.704121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3422.708914] Lustre: 125592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3424.553499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3424.563141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3424.645358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3428.861813] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3437.820318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3437.980247] Lustre: lustre-OST0001: new disk, initializing [ 3437.983528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3437.989575] Lustre: 126461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3439.617122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3439.629572] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3439.751701] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3444.273400] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3453.848182] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3457.444709] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3788.664566] Lustre: 132865:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3788.679403] Lustre: 132865:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3788.690497] Lustre: 132865:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3788.699332] Lustre: 132865:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3788.703407] Lustre: 132865:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3788.713133] Lustre: 132865:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4036.778930] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:25:39 (1786677939) [ 4051.940059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.947108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4051.955938] Lustre: Skipped 3 previous similar messages [ 4057.072059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4057.085963] Lustre: Skipped 6 previous similar messages [ 4057.633831] Lustre: server umount lustre-MDT0000 complete [ 4060.952733] LustreError: 132900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786677964 with bad export cookie 4770857400584233775 [ 4060.960453] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4060.963779] LustreError: 132900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4062.178817] LustreError: 127258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4062.188920] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.209955] LustreError: 127258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4062.259859] Lustre: Skipped 3 previous similar messages [ 4065.184820] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4065.199319] Lustre: Skipped 3 previous similar messages [ 4067.587367] Lustre: server umount lustre-MDT0001 complete [ 4085.215261] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4085.340965] Lustre: server umount lustre-OST0000 complete [ 4095.852707] Lustre: server umount lustre-OST0001 complete [ 4102.629765] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 4114.440345] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4151.146649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.341708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4151.370562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4151.516279] Lustre: lustre-MDT0000: new disk, initializing [ 4151.666189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.670495] Lustre: Skipped 5 previous similar messages [ 4151.693042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4155.731270] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4165.741252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4165.926144] Lustre: 135913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4165.985525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4165.992707] Lustre: Skipped 1 previous similar message [ 4166.065342] Lustre: lustre-MDT0001: new disk, initializing [ 4166.215691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4166.238987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4170.739743] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4180.516769] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4186.070393] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4192.850292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.267827] Lustre: lustre-OST0000: new disk, initializing [ 4193.274939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4193.285282] Lustre: 137884:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4194.727689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4194.742621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4194.962465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4199.953355] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4208.503366] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4218.024819] Lustre: lustre-MDD0000: changelog on [ 4220.773743] Lustre: lustre-MDD0001: changelog on [ 5876.236884] Lustre: lustre-MDD0001: changelog off [ 5878.138373] Lustre: 137080:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5878.846240] Lustre: lustre-MDD0000: changelog off [ 5884.897728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5884.904724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5884.917870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5889.505652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5889.521745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5889.522746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.555598] Lustre: Skipped 1 previous similar message [ 5894.639588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5894.650214] Lustre: Skipped 1 previous similar message [ 5899.233217] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5899.478620] Lustre: server umount lustre-OST0000 complete [ 5904.865634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.867498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5904.887769] Lustre: Skipped 1 previous similar message [ 5909.965243] Lustre: server umount lustre-MDT0000 complete [ 5909.989991] LustreError: 135925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.030342] LustreError: 135925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5914.044024] LustreError: 135906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786679818 with bad export cookie 4770857400585050976 [ 5914.056515] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5914.066500] LustreError: 135906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5914.823267] Lustre: server umount lustre-MDT0001 complete [ 5927.185409] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 5931.295298] Key type lgssc unregistered [ 5931.680287] LNet: 147000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5931.690075] LNetError: 147000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5931.709298] LNet: Removed LNI 192.168.202.115@tcp [ 5932.844151] Key type .llcrypt unregistered [ 5932.847296] Key type ._llcrypt unregistered [ 5956.819468] Key type ._llcrypt registered [ 5956.828514] Key type .llcrypt registered [ 5957.030418] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 5973.644353] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5974.793118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5975.045977] alg: No test for adler32 (adler32-zlib) [ 5976.107297] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5976.334875] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 5977.992441] Key type lgssc registered [ 5979.100750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6016.140772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6016.193448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.446350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6017.484316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6017.579603] Lustre: lustre-MDT0000: new disk, initializing [ 6017.642479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6017.654343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6022.483239] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6034.113786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.266854] Lustre: 150204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6034.348179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6034.362703] Lustre: Skipped 1 previous similar message [ 6034.461934] Lustre: lustre-MDT0001: new disk, initializing [ 6034.551616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6034.588727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6034.604476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6039.626391] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6049.142243] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6055.076597] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6062.590806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.826933] Lustre: lustre-OST0000: new disk, initializing [ 6062.831291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6062.837909] Lustre: 152173:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6062.924867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6070.120546] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6070.904339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6070.927970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6071.117501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6079.872340] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6086.863855] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_module obdecho/obdecho [ 6089.156381] LustreError: 153176:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6089.187295] LustreError: 153176:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6091.635487] LustreError: 153272:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6106.083375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6106.090458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6106.113700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6107.104361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6107.116914] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6107.119791] Lustre: Skipped 1 previous similar message [ 6107.153427] Lustre: Skipped 1 previous similar message [ 6111.434267] Lustre: server umount lustre-MDT0000 complete [ 6115.080122] LustreError: 150197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786680019 with bad export cookie 7891205645431830189 [ 6115.081187] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6115.089062] LustreError: 150197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6115.315955] Lustre: server umount lustre-MDT0001 complete [ 6129.603728] Lustre: server umount lustre-OST0000 complete [ 6153.035573] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 6167.440524] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6209.278616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6209.541629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6209.580753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6209.681245] Lustre: lustre-MDT0000: new disk, initializing [ 6209.767608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6209.789377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6216.498199] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6226.913549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.048414] Lustre: 157024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6227.118873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6227.124802] Lustre: Skipped 1 previous similar message [ 6227.213570] Lustre: lustre-MDT0001: new disk, initializing [ 6227.322270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6227.358239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6227.377785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6232.834408] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6242.902335] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6249.180546] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6257.113921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6257.318162] Lustre: lustre-OST0000: new disk, initializing [ 6257.322089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6257.327572] Lustre: 158993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6257.393239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6264.418516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6264.435707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6264.685589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6264.759569] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6276.017502] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6282.079746] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6282.377530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6288.207606] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6288.484954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6295.526427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6295.532223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.538474] LustreError: Skipped 1 previous similar message [ 6295.540995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6295.541023] Lustre: Skipped 1 previous similar message [ 6295.579824] Lustre: Skipped 1 previous similar message [ 6296.958460] Lustre: server umount lustre-OST0000 complete [ 6304.229083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6304.236980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.247405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6304.258415] Lustre: Skipped 1 previous similar message [ 6307.475973] Lustre: server umount lustre-MDT0000 complete [ 6310.880994] LustreError: 157035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.909731] LustreError: 157035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6311.772036] LustreError: 158992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786680215 with bad export cookie 7891205645431831148 [ 6311.779498] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6311.796427] LustreError: 158992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6312.052757] Lustre: server umount lustre-MDT0001 complete [ 6316.491514] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:03:39 (1786680219) [ 6318.814404] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6320.829530] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:03:43 (1786680223) [ 6329.484404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.065518] LustreError: 161405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.169632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.121329] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6335.457224] LustreError: 161406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6340.584224] LustreError: 161405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6344.076229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.153454] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6358.379130] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6364.341530] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6370.887948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6371.219325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6371.224231] Lustre: Skipped 1 previous similar message [ 6378.015948] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6385.572794] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6399.977778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6399.980501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6400.008441] Lustre: Skipped 3 previous similar messages [ 6400.026863] Lustre: Skipped 4 previous similar messages [ 6403.813635] Lustre: server umount lustre-MDT0000 complete [ 6405.096667] LustreError: 161405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.115116] LustreError: 161405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6407.943811] LustreError: 161387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786680311 with bad export cookie 7891205645431832030 [ 6407.945419] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6407.953781] LustreError: 161387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6410.209591] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.225871] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6410.232710] Lustre: Skipped 1 previous similar message [ 6414.570295] Lustre: server umount lustre-MDT0001 complete [ 6422.832756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6438.063643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6438.410779] LustreError: 165248:0:(llog.c:1655:llog_backup()) MGC192.168.202.115@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6438.420455] Lustre: 165248:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.115@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6441.787521] LustreError: 165253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6441.804191] LustreError: 165253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6458.107997] LustreError: 165254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6458.128194] LustreError: 165254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6492.640641] LustreError: 165254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6492.650991] LustreError: 165254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 6499.299287] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6499.306646] Lustre: 165248:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6540.299173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6545.381928] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6545.433217] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6555.540769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6557.672251] LustreError: 165253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6557.710079] LustreError: 165253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 47 previous similar messages [ 6562.792194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6577.129215] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6d8331b9a1859737 [ 6577.144519] Lustre: MGC192.168.202.115@tcp: Connection restored to 0@lo (at 0@lo) [ 6582.170291] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6582.772169] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6589.336467] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6590.612642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6595.975399] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6597.240858] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6618.593959] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.596713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6618.629048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6618.645022] Lustre: Skipped 3 previous similar messages [ 6620.351755] Lustre: server umount lustre-MDT0000 complete [ 6625.201748] LustreError: 163986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786680529 with bad export cookie 7891205645431838519 [ 6625.206719] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6625.214138] LustreError: 163986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6625.842481] Lustre: server umount lustre-MDT0001 complete [ 6639.798213] Lustre: server umount lustre-OST0000 complete [ 6663.327254] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 6674.583540] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6712.439103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.720456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6712.778434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6712.889744] Lustre: lustre-MDT0000: new disk, initializing [ 6712.965052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6712.976622] Lustre: Skipped 1 previous similar message [ 6713.006405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6718.418710] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6729.343015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.512927] Lustre: 170358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6729.891366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6734.110955] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6744.992565] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6751.217821] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6757.277506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6757.456705] Lustre: lustre-OST0000: new disk, initializing [ 6757.459064] Lustre: Skipped 1 previous similar message [ 6757.461459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6757.464856] Lustre: Skipped 2 previous similar messages [ 6757.469196] Lustre: 172326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6763.858332] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6764.659199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6764.665500] Lustre: Skipped 1 previous similar message [ 6764.667086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6764.770942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6775.621252] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6782.452737] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6782.727358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6790.431445] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6790.911710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6795.753386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6795.761294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6795.785054] Lustre: Skipped 2 previous similar messages [ 6795.799948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6799.432829] Lustre: server umount lustre-OST0000 complete [ 6810.089316] Lustre: server umount lustre-MDT0000 complete [ 6811.111295] LustreError: 170366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6811.139149] LustreError: 170366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 6814.984824] LustreError: 170351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786680718 with bad export cookie 7891205645431855326 [ 6814.989157] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6815.005731] LustreError: 170351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6815.910467] Lustre: server umount lustre-MDT0001 complete [ 6824.932630] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:12:07 (1786680727) [ 6832.162334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.091527] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6854.130386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.904627] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6868.692460] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6874.737177] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6881.137484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6888.586479] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6897.659879] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6919.017728] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:13:41 (1786680821) [ 6924.774536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6924.774751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6924.801818] Lustre: Skipped 3 previous similar messages [ 6924.803263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6924.803408] Lustre: Skipped 3 previous similar messages [ 6924.825400] LustreError: Skipped 1 previous similar message [ 6929.444111] Lustre: server umount lustre-OST0000 complete [ 6939.912486] Lustre: server umount lustre-MDT0000 complete [ 6944.112576] LustreError: 174721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786680848 with bad export cookie 7891205645431856201 [ 6944.121511] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6944.127520] LustreError: 174721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6944.515222] Lustre: server umount lustre-MDT0001 complete [ 6956.033109] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 6959.185191] Key type lgssc unregistered [ 6959.503859] LNet: 178502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6959.524976] LNetError: 178502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6960.550324] LNet: Removed LNI 192.168.202.115@tcp [ 6961.547961] Key type .llcrypt unregistered [ 6961.550441] Key type ._llcrypt unregistered [ 6987.095665] Key type ._llcrypt registered [ 6987.098194] Key type .llcrypt registered [ 6987.190458] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6988.114941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6988.128171] alg: No test for adler32 (adler32-zlib) [ 6989.325659] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6989.676827] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 6991.495171] Key type lgssc registered [ 6992.943189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7001.694789] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7001.708965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.252860] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7022.367614] LustreError: 179745:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.115@tcp: failed processing log, type 4: rc = -110 [ 7027.487578] LustreError: 179745:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7027.501614] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7027.514116] LustreError: MGC192.168.202.115@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7027.533023] LustreError: 179745:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7027.548738] LustreError: 179745:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7027.556573] LustreError: 179745:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7027.674346] Lustre: server umount lustre-OST0000 complete [ 7027.676436] LustreError: 179745:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7036.781909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.469507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7043.682580] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7048.682956] LustreError: 180245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.711925] LustreError: 180245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7053.289599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.682821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7058.753097] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7067.773516] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7075.399827] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7083.707993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7084.025213] LustreError: 182062:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7084.038330] Lustre: 182062:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.115@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7084.238850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.881639] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7099.969300] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7102.224855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7102.243698] Lustre: Skipped 2 previous similar messages [ 7102.248175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7102.259056] Lustre: Skipped 2 previous similar messages [ 7107.554752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7107.572704] Lustre: Skipped 2 previous similar messages [ 7108.761733] Lustre: server umount lustre-MDT0000 complete [ 7112.674506] LustreError: 180242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.701210] LustreError: 180242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7112.722214] LustreError: 180225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786681016 with bad export cookie 1870914866202065145 [ 7112.729184] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7112.732783] LustreError: 180225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7114.208487] LustreError: 180974:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005d92e14d x1873470704087424/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7114.429234] Lustre: server umount lustre-MDT0001 complete [ 7129.639878] Lustre: server umount lustre-OST0000 complete [ 7137.927627] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 7150.001534] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7188.544619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.917662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7188.939478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7188.991540] Lustre: lustre-MDT0000: new disk, initializing [ 7189.087123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7189.106298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7194.068727] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7205.957084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.102899] Lustre: 185590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7206.181513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7206.187170] Lustre: Skipped 1 previous similar message [ 7206.285674] Lustre: lustre-MDT0001: new disk, initializing [ 7206.390851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7206.453555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7206.460450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7211.393841] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7221.767235] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7227.375903] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7234.781653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7235.008915] Lustre: lustre-OST0000: new disk, initializing [ 7235.012643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7235.026425] Lustre: 187555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7235.096506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7242.345865] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7242.820491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7242.831766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7242.972122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7253.702350] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7259.913788] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7260.179638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7265.487980] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7265.674386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7268.834527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.838195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7268.854643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7268.860068] Lustre: Skipped 1 previous similar message [ 7273.959399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.967018] Lustre: Skipped 1 previous similar message [ 7274.284302] Lustre: server umount lustre-OST0000 complete [ 7283.169427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7283.181934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.207689] Lustre: Skipped 1 previous similar message [ 7283.219094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7287.174691] Lustre: server umount lustre-MDT0000 complete [ 7289.314961] LustreError: 186681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.815253] LustreError: 187554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786681197 with bad export cookie 1870914866202065859 [ 7293.818665] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7293.824964] LustreError: 187554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7294.720527] Lustre: server umount lustre-MDT0001 complete [ 7304.228961] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:20:07 (1786681207) [ 7324.329898] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 7327.700691] Key type lgssc unregistered [ 7328.014395] LNet: 190506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7328.024316] LNetError: 190506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7329.066699] LNet: Removed LNI 192.168.202.115@tcp [ 7330.204357] Key type .llcrypt unregistered [ 7330.206785] Key type ._llcrypt unregistered [ 7349.401588] Key type ._llcrypt registered [ 7349.403945] Key type .llcrypt registered [ 7349.495669] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7350.880783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7351.093389] alg: No test for adler32 (adler32-zlib) [ 7352.304836] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7352.737113] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 7354.417731] Key type lgssc registered [ 7355.387702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7366.211691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7366.238159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7368.027454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7373.898650] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7377.378669] LustreError: 191725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7377.402566] LustreError: 191725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7382.501123] LustreError: 191726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7383.421389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.897599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7389.530253] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7398.111286] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7405.458287] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7414.704517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7415.133777] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7415.147246] LustreError: 193593:0:(llog.c:1696:llog_backup()) MGC192.168.202.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7415.165422] Lustre: 193593:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.115@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7415.360551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7422.115363] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7430.717331] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7433.703405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.706316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7433.721448] Lustre: Skipped 2 previous similar messages [ 7433.729321] Lustre: Skipped 1 previous similar message [ 7438.832140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.848445] Lustre: Skipped 3 previous similar messages [ 7439.567577] Lustre: server umount lustre-MDT0000 complete [ 7443.948712] LustreError: 191720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.979764] LustreError: 191707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786681347 with bad export cookie 17423532617278621732 [ 7443.983676] LustreError: 191720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7443.988482] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7443.994507] LustreError: 191707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7444.235953] Lustre: server umount lustre-MDT0001 complete [ 7458.760727] Lustre: server umount lustre-OST0000 complete [ 7470.654978] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 7483.606618] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7525.442179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7525.686038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7525.716787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7525.781528] Lustre: lustre-MDT0000: new disk, initializing [ 7525.891138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7525.944121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7533.412649] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7548.388480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7548.533930] Lustre: 197122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7548.594587] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7548.598199] Lustre: Skipped 1 previous similar message [ 7548.601965] LustreError: 197905:0:(llog.c:1696:llog_backup()) MGC192.168.202.115@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7548.608888] LustreError: 197905:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7548.611771] Lustre: 197905:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.115@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7548.627784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7548.632587] Lustre: Skipped 1 previous similar message [ 7548.709937] Lustre: lustre-MDT0001: new disk, initializing [ 7548.880980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7548.921941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7548.938620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7553.752329] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7564.514751] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7571.215823] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7579.581789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7579.812157] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7579.816166] Lustre: Skipped 1 previous similar message [ 7579.818826] LustreError: 199071:0:(llog.c:1696:llog_backup()) MGC192.168.202.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7579.823790] LustreError: 199071:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7579.829993] Lustre: 199071:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.115@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7579.838395] Lustre: 199071:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7579.861415] Lustre: lustre-OST0000: new disk, initializing [ 7579.864194] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7579.871111] Lustre: 199095:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7579.937725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7587.440623] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7587.899022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7587.907053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7588.008351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7597.995527] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7603.751638] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7603.945447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7609.772560] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7610.016949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7613.924138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7613.926997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7613.941937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7613.966042] Lustre: Skipped 1 previous similar message [ 7613.989334] Lustre: Skipped 1 previous similar message [ 7618.528065] Lustre: server umount lustre-OST0000 complete [ 7624.167768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.198092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7624.200044] Lustre: Skipped 1 previous similar message [ 7624.217677] Lustre: Skipped 1 previous similar message [ 7629.288870] LustreError: 197133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7629.331252] Lustre: server umount lustre-MDT0000 complete [ 7633.379361] LustreError: 197115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786681537 with bad export cookie 17423532617278622446 [ 7633.395628] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7633.398905] LustreError: 197115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7633.725413] Lustre: server umount lustre-MDT0001 complete [ 7643.481979] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:25:46 (1786681546) [ 7662.302405] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7673.055896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.540485] LustreError: 202221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.556416] LustreError: 202221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7673.581774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7678.879481] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7683.043162] LustreError: 202222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.071917] LustreError: 202222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7687.403457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.108772] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7697.623034] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7707.098980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7707.704463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7707.709282] Lustre: Skipped 1 previous similar message [ 7716.488220] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7726.436165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7726.719387] Lustre: lustre-OST0001: new disk, initializing [ 7726.724970] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7726.741823] Lustre: 204532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7734.485547] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7734.944248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7734.954136] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7735.134246] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7747.161184] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7757.063606] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7759.631784] Lustre: DEBUG MARKER: ADD OST3 [ 7766.199870] loop: module loaded [ 7766.294055] loop0: detected capacity change from 0 to 204800000 [ 7766.718561] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7766.733678] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7766.750504] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7774.469327] loop0: detected capacity change from 0 to 204800000 [ 7774.878844] Lustre: DEBUG MARKER: STOP OST3 [ 7782.182540] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7782.366399] Lustre: lustre-OST0003: new disk, initializing [ 7782.371488] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7782.380446] Lustre: 206571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7782.459273] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7782.461817] Lustre: Skipped 1 previous similar message [ 7783.987563] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7784.258854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7784.264823] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7784.325464] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7786.016085] LustreError: 206698:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7786.593643] Lustre: Failing over lustre-OST0003 [ 7786.843890] Lustre: server umount lustre-OST0003 complete [ 7789.427156] LustreError: 203710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7789.557574] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.558437] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.528321] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7801.165714] Lustre: DEBUG MARKER: START OST3 again [ 7808.734294] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7809.088069] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7809.808796] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7811.096377] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 7811.099448] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7815.754956] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7836.926522] Lustre: Failing over lustre-OST0003 [ 7837.040594] Lustre: server umount lustre-OST0003 complete [ 7837.160089] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7837.165563] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.193146] Lustre: Skipped 1 previous similar message [ 7837.203065] LustreError: 204916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.250993] LustreError: 204916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7860.197150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.214582] Lustre: Skipped 1 previous similar message [ 7860.223911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7860.230953] Lustre: Skipped 2 previous similar messages [ 7864.225981] Lustre: server umount lustre-MDT0000 complete [ 7868.269770] LustreError: 206567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786681772 with bad export cookie 17423532617278623328 [ 7868.279703] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7868.283098] LustreError: 206567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7870.453824] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7870.461382] Lustre: Skipped 1 previous similar message [ 7874.918290] Lustre: server umount lustre-MDT0001 complete [ 7878.209561] Lustre: server umount lustre-OST0000 complete [ 7882.313293] Lustre: server umount lustre-OST0001 complete [ 7890.763711] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 7899.461712] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7937.888796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7938.165088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7938.196430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7938.302481] Lustre: lustre-MDT0000: new disk, initializing [ 7938.407658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.415030] Lustre: Skipped 1 previous similar message [ 7938.427830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7943.420517] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7954.210542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7954.346690] Lustre: 211995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7954.410598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7954.415136] Lustre: Skipped 1 previous similar message [ 7954.497694] Lustre: lustre-MDT0001: new disk, initializing [ 7954.673053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7954.691908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7959.226644] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7970.779838] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7977.507847] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7985.082165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7985.522431] Lustre: 213964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7986.986458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7986.997663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7987.141732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7992.605788] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8004.135462] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8009.763141] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8009.957898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8015.114559] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8015.374820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8017.888823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8017.910283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8017.910618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.939784] Lustre: Skipped 2 previous similar messages [ 8017.972996] Lustre: Skipped 6 previous similar messages [ 8023.833310] Lustre: server umount lustre-OST0000 complete [ 8034.214271] Lustre: server umount lustre-MDT0000 complete [ 8038.370853] LustreError: 212000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8038.386609] LustreError: 212000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 8038.470551] LustreError: 212788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786681942 with bad export cookie 17423532617278625624 [ 8038.482899] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8038.490723] LustreError: 212788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8038.845743] Lustre: server umount lustre-MDT0001 complete [ 8042.279411] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:32:25 (1786681945) [ 8059.143881] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 8070.686703] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8110.689991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8111.063618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8111.141455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8111.151571] Lustre: Skipped 1 previous similar message [ 8111.295407] Lustre: lustre-MDT0000: new disk, initializing [ 8111.300265] Lustre: Skipped 1 previous similar message [ 8111.482268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8117.142728] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8129.757882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8129.843559] Lustre: 218574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8130.153731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8135.986105] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8146.735765] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8152.925203] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8159.399958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8159.717981] Lustre: 220546:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8162.043208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8168.483768] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8179.655992] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8184.872685] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8185.153631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8190.369557] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8190.591666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8197.604513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8197.610610] LustreError: Skipped 1 previous similar message [ 8197.614217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8197.636749] Lustre: Skipped 2 previous similar messages [ 8197.644898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8197.651377] Lustre: Skipped 7 previous similar messages [ 8199.777708] Lustre: server umount lustre-OST0000 complete [ 8206.816774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8206.821539] LustreError: Skipped 1 previous similar message [ 8210.918431] Lustre: server umount lustre-MDT0000 complete [ 8213.477711] LustreError: 221217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8213.477711] LustreError: 218585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8215.104463] LustreError: 218564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786682119 with bad export cookie 17423532617278626506 [ 8215.111873] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8215.121940] LustreError: 218564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8215.811462] Lustre: server umount lustre-MDT0001 complete [ 8225.173628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.663926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.668945] Lustre: Skipped 5 previous similar messages [ 8230.264652] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8240.045656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8245.361522] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8253.930185] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8260.055960] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8268.335502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8275.523996] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8284.593890] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8297.442805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8297.460358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8297.476922] Lustre: Skipped 3 previous similar messages [ 8297.483388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8297.492020] Lustre: Skipped 5 previous similar messages [ 8301.953252] Lustre: server umount lustre-OST0000 complete [ 8312.474896] Lustre: server umount lustre-MDT0000 complete [ 8316.798301] LustreError: 222745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786682220 with bad export cookie 17423532617278627388 [ 8316.806391] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8316.822987] LustreError: 222745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8317.630353] Lustre: server umount lustre-MDT0001 complete [ 8426.251302] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 8428.946328] Key type lgssc unregistered [ 8429.230734] LNet: 226435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8429.235804] LNetError: 226435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8430.264058] LNet: Removed LNI 192.168.202.115@tcp [ 8431.332242] Key type .llcrypt unregistered [ 8431.334872] Key type ._llcrypt unregistered [ 8457.558836] Key type ._llcrypt registered [ 8457.561273] Key type .llcrypt registered [ 8457.677849] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 8477.106086] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8478.214342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8478.241844] alg: No test for adler32 (adler32-zlib) [ 8479.375565] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 8479.588024] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 8481.295317] Key type lgssc registered [ 8482.959627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8524.115424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8524.164643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8525.592241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8525.633229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8525.746222] Lustre: lustre-MDT0000: new disk, initializing [ 8525.906781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8525.939188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8530.816491] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8542.329607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.428683] Lustre: 229793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8542.489232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8542.496957] Lustre: Skipped 1 previous similar message [ 8542.596683] Lustre: lustre-MDT0001: new disk, initializing [ 8542.759269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8542.822394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8542.841323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8547.029624] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8555.988305] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8560.885870] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8566.913314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8567.203368] Lustre: lustre-OST0000: new disk, initializing [ 8567.208496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8567.216226] Lustre: 231759:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8567.355937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8573.365662] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8575.040807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8575.048797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8575.174600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8582.767514] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8587.142985] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8587.357619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8593.546683] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8593.808624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8601.057836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8601.060449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8601.087992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8601.108080] Lustre: Skipped 1 previous similar message [ 8601.134467] Lustre: Skipped 1 previous similar message [ 8602.405903] Lustre: server umount lustre-OST0000 complete [ 8609.250343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8609.259676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8609.286881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8611.296955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8611.298377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8611.317972] Lustre: Skipped 1 previous similar message [ 8613.510517] Lustre: server umount lustre-MDT0000 complete [ 8617.363949] LustreError: 229785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786682521 with bad export cookie 17494210389338457579 [ 8617.372164] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8617.386560] LustreError: 229785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8618.191978] Lustre: server umount lustre-MDT0001 complete [ 8640.411381] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8652.855219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8653.609067] LustreError: 234693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8653.650792] LustreError: 234693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8653.751451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8658.914662] LustreError: 234694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8660.541137] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8663.014901] LustreError: 234693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8668.142951] LustreError: 234694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8671.568634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8672.203144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8678.093426] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8681.578246] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8687.891425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8688.609867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8696.129466] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8704.737796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8704.915419] Lustre: lustre-OST0001: new disk, initializing [ 8704.918397] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8704.927061] Lustre: 237002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8705.102042] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8710.779644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8710.797699] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8710.951753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8712.012709] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8724.366723] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8732.332591] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8745.914740] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:44:08 (1786682648) [ 8749.026175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8749.031585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8749.042587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8754.834227] Lustre: server umount lustre-MDT0000 complete [ 8755.191769] LustreError: 235422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8755.212085] LustreError: 235422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8758.759647] LustreError: 235429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786682662 with bad export cookie 17494210389338458461 [ 8758.761340] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8758.768994] LustreError: 235429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8759.088095] Lustre: server umount lustre-MDT0001 complete [ 8773.590881] Lustre: server umount lustre-OST0000 complete [ 8776.480264] Lustre: 228017:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786682664/real 1786682664] req@ffff8b30d0877b80 x1873472266551936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786682680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8776.510471] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.524130] Lustre: Skipped 3 previous similar messages [ 8777.861115] Lustre: server umount lustre-OST0001 complete [ 8785.583596] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 8796.780702] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8837.920636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.245967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8838.296537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8838.370137] Lustre: lustre-MDT0000: new disk, initializing [ 8838.427643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8838.441936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8844.139427] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8856.286715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8856.447723] Lustre: 240950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8856.500361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8856.504523] Lustre: Skipped 1 previous similar message [ 8856.591836] Lustre: lustre-MDT0001: new disk, initializing [ 8856.815719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8856.827793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8862.514289] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8872.951617] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8879.245824] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8885.544388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8885.798916] Lustre: lustre-OST0000: new disk, initializing [ 8885.804884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8885.811677] Lustre: 242918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8887.918087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8887.928658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8888.091688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8892.799882] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8903.088805] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8908.432479] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8908.706438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8912.565883] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8912.765304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8919.015243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8919.027854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8919.049482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8919.054646] Lustre: Skipped 4 previous similar messages [ 8920.659540] Lustre: server umount lustre-OST0000 complete [ 8928.224816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8928.236804] LustreError: Skipped 1 previous similar message [ 8928.240934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8930.718267] Lustre: server umount lustre-MDT0000 complete [ 8934.076428] LustreError: 240941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786682838 with bad export cookie 17494210389338460596 [ 8934.101467] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8934.444996] Lustre: server umount lustre-MDT0001 complete [ 8943.635669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8944.082331] LustreError: 245134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8944.170191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8944.173470] Lustre: Skipped 2 previous similar messages [ 8949.024302] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8956.838681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8962.395486] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8969.995185] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8977.483086] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8984.770875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8989.838678] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8997.731135] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9142.752113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9142.756057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9142.768995] Lustre: Skipped 3 previous similar messages [ 9142.783633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9142.793499] Lustre: Skipped 2 previous similar messages [ 9147.932298] Lustre: server umount lustre-OST0000 complete [ 9168.354356] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9168.882277] Lustre: server umount lustre-MDT0000 complete [ 9170.403958] LustreError: 247645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9170.430197] LustreError: 247645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9173.491906] LustreError: 245116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786683077 with bad export cookie 17494210389338461478 [ 9173.493582] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9173.506162] LustreError: 245116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9173.803345] Lustre: server umount lustre-MDT0001 complete [ 9267.092383] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9270.511953] Key type lgssc unregistered [ 9270.756510] LNet: 248744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9270.765579] LNetError: 248744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9270.780712] LNet: Removed LNI 192.168.202.115@tcp [ 9271.740148] Key type .llcrypt unregistered [ 9271.742434] Key type ._llcrypt unregistered [ 9303.813525] Key type ._llcrypt registered [ 9303.816959] Key type .llcrypt registered [ 9303.989372] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 9320.775783] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9322.193790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9322.359643] alg: No test for adler32 (adler32-zlib) [ 9323.400965] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 9323.690200] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 9325.351235] Key type lgssc registered [ 9326.692947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9371.319723] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9382.250753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9382.305725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9383.733092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9383.808130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9383.918426] Lustre: lustre-MDT0000: new disk, initializing [ 9384.007428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.035156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9389.687962] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9402.908024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.082976] Lustre: 252819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9403.145934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9403.149274] Lustre: Skipped 1 previous similar message [ 9403.219685] Lustre: lustre-MDT0001: new disk, initializing [ 9403.284413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9403.307892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9403.327065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9407.391578] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9412.352805] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9419.364966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9419.811778] Lustre: lustre-OST0000: new disk, initializing [ 9419.815614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9419.821686] Lustre: 254451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9419.828093] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 9419.904660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.485365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9425.507985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9425.559918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 9427.850913] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9439.300911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9439.554286] Lustre: lustre-OST0001: new disk, initializing [ 9439.557248] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9439.561993] Lustre: 255320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9439.667123] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9446.244462] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9446.567888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9446.584277] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9446.725264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9458.200792] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9465.530176] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9477.607488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9477.620462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9477.630837] Lustre: Skipped 1 previous similar message [ 9482.741061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9482.745092] Lustre: Skipped 3 previous similar messages [ 9487.844840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9487.856667] Lustre: Skipped 2 previous similar messages [ 9490.400049] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9490.724305] Lustre: server umount lustre-MDT0000 complete [ 9495.559745] LustreError: 253621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786683399 with bad export cookie 12743345013667128758 [ 9495.566935] LustreError: 253621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9495.572929] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9495.818742] Lustre: server umount lustre-MDT0001 complete [ 9509.221192] Lustre: server umount lustre-OST0000 complete [ 9522.354531] Lustre: server umount lustre-OST0001 complete [ 9535.680385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9546.357508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9554.971222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9568.501839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9597.442412] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9608.817085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.852820] Lustre: lustre-MDT0000: reset Object Index mappings [ 9609.472106] LustreError: 260332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9609.505276] LustreError: 260332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9609.586083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9614.604444] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9614.826407] LustreError: 260333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9619.942443] LustreError: 260332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9625.063416] LustreError: 260333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9628.658367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9628.728571] Lustre: lustre-MDT0001: reset Object Index mappings [ 9630.167519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9638.521857] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9643.730069] Lustre: 261476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9652.222696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9652.929122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9654.981839] LustreError: 261830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9655.035311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 9655.037919] LustreError: 261830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9660.551204] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9665.512373] LustreError: 261959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9665.528755] LustreError: 261959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9670.586658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9676.288844] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9677.811385] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9685.355768] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9688.903823] Lustre: 263349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9696.060695] Lustre: server umount lustre-MDT0000 complete [ 9696.224926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9696.234781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.249792] Lustre: Skipped 3 previous similar messages [ 9696.257413] LustreError: 261170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9696.273098] LustreError: 261170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9700.144785] LustreError: 261856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786683604 with bad export cookie 12743345013667135548 [ 9700.149399] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9700.159089] LustreError: 261856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9700.797137] Lustre: server umount lustre-MDT0001 complete [ 9716.039627] Lustre: server umount lustre-OST0000 complete [ 9717.728100] Lustre: 250327:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786683605/real 1786683605] req@ffff8b31c8ecd500 x1873473151565184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786683621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9717.784799] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9717.820989] Lustre: Skipped 3 previous similar messages [ 9721.698867] Lustre: 250325:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786683609/real 1786683609] req@ffff8b31e61ee300 x1873473151565440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786683625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9723.148975] Lustre: server umount lustre-OST0001 complete [ 9751.311908] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:00:53 (1786683653) [ 9772.285084] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 9784.727761] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9826.318561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9826.615393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9826.650739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9826.747889] Lustre: lustre-MDT0000: new disk, initializing [ 9826.811933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9826.816974] Lustre: Skipped 1 previous similar message [ 9826.830739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9831.469903] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9843.790678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.904580] Lustre: 267620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9843.957551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9843.960671] Lustre: Skipped 1 previous similar message [ 9844.017562] Lustre: lustre-MDT0001: new disk, initializing [ 9844.191136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9844.200630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9849.001868] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9858.853395] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9865.342621] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9873.041665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9873.284368] Lustre: lustre-OST0000: new disk, initializing [ 9873.287661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9873.300516] Lustre: 269589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9874.682669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9874.694264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9874.891352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9880.817555] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9890.849251] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9896.389912] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9896.707103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9902.402397] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9902.671299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9905.639763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9905.664995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9905.666047] Lustre: Skipped 1 previous similar message [ 9905.667192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9905.702979] Lustre: Skipped 2 previous similar messages [ 9910.927284] Lustre: server umount lustre-OST0000 complete [ 9915.872311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9915.878884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9915.885928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9915.885938] Lustre: Skipped 2 previous similar messages [ 9915.912158] Lustre: Skipped 1 previous similar message [ 9920.997278] LustreError: 267632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9921.013966] LustreError: 267632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9921.051159] Lustre: server umount lustre-MDT0000 complete [ 9925.604322] LustreError: 267613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786683829 with bad export cookie 12743345013667137641 [ 9925.613049] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9925.626816] LustreError: 267613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9926.288564] Lustre: server umount lustre-MDT0001 complete [ 9952.027278] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9965.072478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9965.492745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.501535] Lustre: Skipped 2 previous similar messages [ 9970.981387] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9980.418645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9986.509663] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9989.829621] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9997.694419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10006.162662] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10015.970425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10016.255114] Lustre: lustre-OST0001: new disk, initializing [10016.265332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10016.270781] Lustre: 274854:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10017.433664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10017.441117] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10017.557774] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10023.692843] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10033.495914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10038.016959] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10043.465324] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10043.891540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10052.577190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10052.590257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10052.616834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10052.619370] Lustre: Skipped 1 previous similar message [10064.351758] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10064.711186] Lustre: server umount lustre-MDT0000 complete [10068.326747] LustreError: 273282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786683972 with bad export cookie 12743345013667138523 [10068.328396] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10068.332737] LustreError: 273282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10068.550757] Lustre: server umount lustre-MDT0001 complete [10083.866528] Lustre: server umount lustre-OST0000 complete [10084.320358] Lustre: 250327:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786683972/real 1786683972] req@ffff8b31c3577b80 x1873473151677184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786683988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10084.348143] Lustre: 250327:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10084.357816] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10084.370670] Lustre: Skipped 3 previous similar messages [10088.218957] Lustre: server umount lustre-OST0001 complete [10106.662926] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10120.315285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10121.024909] LustreError: 277910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10121.043561] LustreError: 277910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10121.140285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10121.144262] Lustre: Skipped 3 previous similar messages [10127.443959] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10135.739335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10140.490910] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10143.788281] Lustre: 279051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10151.186414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10158.580051] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10167.087069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10167.222530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10172.934860] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10175.517732] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10184.559188] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10189.196273] Lustre: 280929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10195.600730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10195.634205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10205.928760] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:08:28 (1786684108) [10223.077953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10223.123509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10223.184077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10223.198884] Lustre: Skipped 12 previous similar messages [10227.438392] Lustre: server umount lustre-MDT0000 complete [10232.239310] LustreError: 277890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786684136 with bad export cookie 12743345013667145089 [10232.246126] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10232.260520] LustreError: 277890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10238.778392] Lustre: server umount lustre-MDT0001 complete [10257.376288] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10257.627878] Lustre: server umount lustre-OST0000 complete [10262.466624] Lustre: server umount lustre-OST0001 complete [10271.931398] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [10286.241115] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10345.544640] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10355.926389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.149305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10356.170678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10356.231753] Lustre: lustre-MDT0000: new disk, initializing [10356.318336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10361.545646] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10372.778780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10373.017417] Lustre: 285173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10373.378548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10377.958463] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10391.737945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10392.047965] Lustre: 286807:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10392.159767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10392.169124] Lustre: Skipped 5 previous similar messages [10393.979303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10399.722315] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10412.911747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10413.097783] Lustre: 287680:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10415.274495] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10415.304995] Lustre: Skipped 1 previous similar message [10415.576757] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10423.646879] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10436.552962] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10440.315148] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10440.318444] Lustre: Skipped 1 previous similar message [10459.807330] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:12:42 (1786684362) [10464.741725] Lustre: server umount lustre-MDT0000 complete [10465.762862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10465.776215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10465.790466] Lustre: Skipped 5 previous similar messages [10465.795730] LustreError: 285185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10465.810883] LustreError: 285185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [10470.211693] LustreError: 285166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786684374 with bad export cookie 12743345013667152768 [10470.223025] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10470.227032] LustreError: 285166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10470.858872] Lustre: server umount lustre-MDT0001 complete [10486.425103] Lustre: server umount lustre-OST0000 complete [10487.264160] Lustre: 250326:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786684375/real 1786684375] req@ffff8b31e567f480 x1873473151833216/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786684391 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10492.536980] Lustre: server umount lustre-OST0001 complete [10503.549253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10507.797351] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10517.806044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10523.049703] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10531.912820] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10539.726874] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10547.646217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10555.812455] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10565.796223] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10583.408314] loop1: detected capacity change from 0 to 204800000 [10583.647364] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10583.657475] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10583.676162] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10589.058172] loop1: detected capacity change from 0 to 204800000 [10595.177694] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10595.283987] LustreError: 289916:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs30997: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10595.291714] LustreError: 289916:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs30997 [10595.295505] LustreError: 289916:0:(mgs_handler.c:517:mgs_target_reg()) fs30997-MDT0001: Error checking target index: rc = -13 [10595.306569] LustreError: fs30997-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10595.312827] LustreError: 293649:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10595.315717] LustreError: 293649:0:(tgt_mount.c:2129:server_put_super()) no obd fs30997-MDT0001 [10595.317930] LustreError: 293649:0:(tgt_mount.c:116:server_deregister_mount()) fs30997-MDT0001 not registered [10595.346519] Lustre: server umount fs30997-MDT0001 complete [10595.349983] LustreError: 293649:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10607.720592] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10607.792405] Lustre: Setting parameter fs30997-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs30997-MDT0001 [10607.863669] Lustre: srv-fs30997-MDT0001: No data found on store. Initialize space. [10607.873255] Lustre: Skipped 4 previous similar messages [10607.997754] Lustre: fs30997-MDT0001: new disk, initializing [10608.000441] Lustre: Skipped 3 previous similar messages [10608.147515] Lustre: srv-fs30997-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10608.159145] Lustre: 294116:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30997-MDT0001-mdtlov: can't create new llog: rc = -115 [10608.170464] LustreError: 294116:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30997-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10615.689510] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10625.361041] Lustre: server umount fs30997-MDT0001 complete [10630.125768] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10641.332588] loop1: detected capacity change from 0 to 204800000 [10648.440955] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.735547] Lustre: 295686:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30997-MDT0001-mdtlov: can't create new llog: rc = -115 [10648.746070] LustreError: 295686:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30997-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10654.675251] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10660.576419] Lustre: server umount fs30997-MDT0001 complete [10673.957768] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.15@tcp (stopping) [10673.967678] Lustre: Skipped 8 previous similar messages [10676.212107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10676.232466] Lustre: Skipped 6 previous similar messages [10678.648231] Lustre: server umount lustre-MDT0000 complete [10689.866637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10690.071760] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10695.691498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10696.101697] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10705.498283] loop1: detected capacity change from 0 to 204800000 [10713.289499] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10714.202145] Lustre: 298133:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30997-MDT0001-mdtlov: can't create new llog: rc = -115 [10714.205140] LustreError: 298133:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30997-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10720.387985] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10725.136037] LustreError: 291762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.175731] LustreError: 291762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 111 previous similar messages [10727.363895] Lustre: server umount fs30997-MDT0001 complete [10751.976496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10751.981534] LustreError: Skipped 1 previous similar message [10756.889643] Lustre: server umount lustre-OST0000 complete [10767.858622] Lustre: server umount lustre-MDT0000 complete [10772.913358] LustreError: 289909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786684676 with bad export cookie 12743345013667155449 [10772.917232] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10772.920575] LustreError: 289909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10773.405436] Lustre: server umount lustre-MDT0001 complete [10786.188446] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [10789.865571] Key type lgssc unregistered [10790.134828] LNet: 300481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10790.140351] LNetError: 300481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10790.161569] LNet: Removed LNI 192.168.202.115@tcp [10791.320630] Key type .llcrypt unregistered [10791.329440] Key type ._llcrypt unregistered [10814.377254] Key type ._llcrypt registered [10814.379717] Key type .llcrypt registered [10814.478980] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10815.867154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10815.904078] alg: No test for adler32 (adler32-zlib) [10817.132433] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [10817.338259] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [10819.080106] Key type lgssc registered [10820.393573] Lustre: Echo OBD driver; http://www.lustre.org/ [10829.520199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10829.548502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10830.912712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10837.435550] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10840.044341] LustreError: 301755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10840.089101] LustreError: 301755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10845.155615] LustreError: 301754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10847.841954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10848.267067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10852.866852] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10861.061839] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10866.726041] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10874.579722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10875.254675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10876.273848] LustreError: 303585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10881.510223] LustreError: 303586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10884.443847] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10886.632705] LustreError: 303643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.261312] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10898.021411] LustreError: 303643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10898.063652] LustreError: 303643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10914.282445] LustreError: 303587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10914.314086] LustreError: 303587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10917.624226] loop1: detected capacity change from 0 to 204800000 [10917.881857] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10917.906030] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10917.918392] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10924.133419] loop1: detected capacity change from 0 to 204800000 [10931.364934] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10931.637387] Lustre: fs5216-OST0000: new disk, initializing [10931.642377] Lustre: srv-fs5216-OST0000: No data found on store. Initialize space. [10931.656745] Lustre: 305494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10931.817804] Lustre: fs5216-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10939.417142] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10949.373065] LustreError: 303586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10949.402748] LustreError: 303586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10949.763707] Lustre: server umount fs5216-OST0000 complete [10955.258335] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10965.423258] loop1: detected capacity change from 0 to 204800000 [10972.359321] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10972.917785] Lustre: fs5216-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10975.736337] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10981.207328] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10991.137744] loop2: detected capacity change from 0 to 204800000 [10991.380963] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10991.389498] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10991.412554] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10996.998678] loop2: detected capacity change from 0 to 204800000 [11003.080043] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11003.159258] LustreError: fs5216-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11003.170403] LustreError: fs5216-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11003.196445] LustreError: 308535:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11003.206073] LustreError: 308535:0:(tgt_mount.c:2129:server_put_super()) no obd fs5216-OST0001 [11003.213336] LustreError: 308535:0:(tgt_mount.c:116:server_deregister_mount()) fs5216-OST0001 not registered [11003.243738] Lustre: server umount fs5216-OST0001 complete [11003.248114] LustreError: 308535:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11015.420609] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11015.580431] Lustre: fs5216-OST0001: new disk, initializing [11015.584440] Lustre: srv-fs5216-OST0001: No data found on store. Initialize space. [11015.594594] Lustre: 309086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11015.706761] Lustre: fs5216-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11015.941263] LustreError: 303586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11015.964302] LustreError: 303586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11023.750890] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11032.579544] Lustre: server umount fs5216-OST0000 complete [11043.113276] Lustre: server umount fs5216-OST0001 complete [11055.791630] loop1: detected capacity change from 0 to 204800000 [11062.313187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11062.560211] Lustre: fs5216-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11070.900866] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11078.927555] loop2: detected capacity change from 0 to 204800000 [11086.610577] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11095.361550] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11107.138954] Lustre: server umount fs5216-OST0000 complete [11108.852261] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11118.206839] Lustre: server umount fs5216-OST0001 complete [11129.330306] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11132.393424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11132.420636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11134.435630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11134.453244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11134.461586] LustreError: Skipped 1 previous similar message [11134.461633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11134.519265] Lustre: Skipped 1 previous similar message [11136.571383] Lustre: server umount lustre-OST0000 complete [11144.683907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11144.698084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11144.712622] Lustre: Skipped 1 previous similar message [11144.719607] Lustre: Skipped 1 previous similar message [11147.096312] Lustre: server umount lustre-MDT0000 complete [11149.795327] LustreError: 301750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.829281] LustreError: 301750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 70 previous similar messages [11152.205494] LustreError: 309085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786685056 with bad export cookie 10115780313137652831 [11152.214828] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11152.215800] LustreError: 309085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11153.323679] Lustre: server umount lustre-MDT0001 complete [11167.362912] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [11170.841688] Key type lgssc unregistered [11171.284209] LNet: 315016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11171.288874] LNetError: 315016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11172.327487] LNet: Removed LNI 192.168.202.115@tcp [11173.431149] Key type .llcrypt unregistered [11173.433785] Key type ._llcrypt unregistered [11207.483329] Key type ._llcrypt registered [11207.486241] Key type .llcrypt registered [11207.640305] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [11225.034703] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [11226.372974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11226.413195] alg: No test for adler32 (adler32-zlib) [11227.440676] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11227.774894] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [11229.551227] Key type lgssc registered [11230.978790] Lustre: Echo OBD driver; http://www.lustre.org/ [11269.332239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11269.354568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11270.779767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11270.797396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11270.866881] Lustre: lustre-MDT0000: new disk, initializing [11270.932910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11270.953268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11276.029037] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11289.171642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11289.299614] Lustre: 318879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11289.343787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11289.346741] Lustre: Skipped 1 previous similar message [11289.397968] Lustre: lustre-MDT0001: new disk, initializing [11289.472629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11289.512223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11289.522636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11294.917054] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11304.206994] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11311.724397] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11319.454315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11319.627544] Lustre: lustre-OST0000: new disk, initializing [11319.629644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11319.633176] Lustre: 320845:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11319.686811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11326.529631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11326.538985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11326.662688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11326.705429] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11336.488201] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11338.979512] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3 [11340.197962] Lustre: Unmounted lustre-client [11352.549684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11352.551520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11352.553981] LustreError: Skipped 1 previous similar message [11352.554952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11352.587424] Lustre: Skipped 1 previous similar message [11354.605833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11354.616696] Lustre: Skipped 1 previous similar message [11356.642078] Lustre: server umount lustre-OST0000 complete [11361.250738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11361.256868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11361.273964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11361.278758] Lustre: Skipped 1 previous similar message [11364.832684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11364.836867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11366.875189] Lustre: server umount lustre-MDT0000 complete [11372.890087] LustreError: 318869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786685276 with bad export cookie 12035645158939236612 [11372.893765] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11372.896921] LustreError: 318869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11373.269579] Lustre: server umount lustre-MDT0001 complete [11387.629311] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [11391.339673] Key type lgssc unregistered [11396.961503] LNet: 323011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11396.985836] LNetError: 323011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11470.773931] LNet: Removed LNI 192.168.202.115@tcp [11471.780339] Key type .llcrypt unregistered [11471.784435] Key type ._llcrypt unregistered [11488.658384] Key type ._llcrypt registered [11488.661760] Key type .llcrypt registered [11488.777310] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [11505.663596] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [11506.954540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11507.113949] alg: No test for adler32 (adler32-zlib) [11508.339538] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11508.753482] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [11510.471191] Key type lgssc registered [11511.666713] Lustre: Echo OBD driver; http://www.lustre.org/ [11546.700079] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:30:49 (1786685449) [11554.697831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11554.747502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11556.398783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11556.473979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11556.657379] Lustre: lustre-MDT0000: new disk, initializing [11556.765609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11556.787408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11562.738855] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11574.340962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11574.412916] Lustre: 326260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11574.443226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11574.445610] Lustre: Skipped 1 previous similar message [11574.484358] Lustre: lustre-MDT0001: new disk, initializing [11574.556213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11574.586930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11574.594405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11579.519158] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11589.162440] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11595.052241] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11601.973762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11602.454604] Lustre: lustre-OST0000: new disk, initializing [11602.460357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11602.472857] Lustre: 328226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11602.589563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11608.594465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11608.612953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11608.758564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11609.792839] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11619.440406] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11634.659577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11634.690090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11639.266843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11639.278483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11639.308993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11640.620159] Lustre: server umount lustre-OST0000 complete [11646.449657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11646.479530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11646.490051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11646.494550] Lustre: Skipped 1 previous similar message [11650.024421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11650.044906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11651.878899] Lustre: server umount lustre-MDT0000 complete [11656.501502] LustreError: 326251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786685560 with bad export cookie 16922844709511817248 [11656.503933] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11656.509985] LustreError: 326251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11656.992346] Lustre: server umount lustre-MDT0001 complete [11671.297771] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [11674.933763] Key type lgssc unregistered [11675.218519] LNet: 330137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11675.228949] LNetError: 330137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11675.255162] LNet: Removed LNI 192.168.202.115@tcp [11676.451945] Key type .llcrypt unregistered [11676.453817] Key type ._llcrypt unregistered [11700.706293] Key type ._llcrypt registered [11700.710512] Key type .llcrypt registered [11700.849338] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [11702.125600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11702.167342] alg: No test for adler32 (adler32-zlib) [11703.234429] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11703.523789] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [11705.184546] Key type lgssc registered [11706.391378] Lustre: Echo OBD driver; http://www.lustre.org/ [11716.167197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11716.201527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11717.717539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11723.921334] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11727.844927] LustreError: 331410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.868363] LustreError: 331410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11732.961084] LustreError: 331409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11733.823352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11734.211225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11739.225318] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11747.591674] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11754.250353] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11761.405256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11761.793866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11768.761663] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11777.455597] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11790.360229] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11792.042946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11799.842049] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.15@tcp (stopping) [11800.045569] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11800.058114] Lustre: Skipped 2 previous similar messages [11804.923482] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.15@tcp (stopping) [11804.928903] Lustre: Skipped 3 previous similar messages [11810.052724] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.15@tcp (stopping) [11810.078280] Lustre: Skipped 3 previous similar messages [11811.295127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11811.621697] Lustre: server umount lustre-MDT0000 complete [11815.170491] LustreError: 331405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11816.202839] LustreError: 331389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786685720 with bad export cookie 9130321508842863745 [11816.213623] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11816.215751] LustreError: 331389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11820.281206] LustreError: 331404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11820.309167] LustreError: 331404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11820.321534] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.15@tcp (stopping) [11820.335855] Lustre: Skipped 3 previous similar messages [11820.515230] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.640574] Lustre: server umount lustre-MDT0001 complete [11827.888752] Lustre: server umount lustre-OST0000 complete [11836.965974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11842.583654] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11845.912406] Lustre: 335596:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.115@tcp [11848.091331] Lustre: server umount MGS complete [11856.808031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11857.515121] LustreError: 336168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11857.526037] LustreError: 336168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11857.574802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11862.561580] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11867.113061] LustreError: 336169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11867.132244] LustreError: 336169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11871.116029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11871.541342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11876.467343] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11884.314527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11884.689153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11891.109349] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11899.327214] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11901.193201] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11907.268962] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11909.161476] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11916.606790] Lustre: server umount lustre-OST0000 complete [11922.914171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11922.922524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11922.923946] Lustre: Skipped 1 previous similar message [11922.938189] Lustre: Skipped 3 previous similar messages [11933.089260] Lustre: 330722:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786685821/real 1786685821] req@00000000d37ced4e x1873475647111552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786685837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11933.131679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.152662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11933.163855] Lustre: Skipped 3 previous similar messages [11935.201642] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11935.493887] Lustre: server umount lustre-MDT0000 complete [11938.274061] Lustre: 330722:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786685826/real 1786685826] req@00000000702da7e2 x1873475647113600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786685842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11938.297668] LustreError: 336168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11938.337440] Lustre: 330722:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11938.370103] LustreError: 336168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11939.713949] LustreError: 336149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786685843 with bad export cookie 9130321508842864585 [11939.733780] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11939.735027] LustreError: 336149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11940.383870] Lustre: server umount lustre-MDT0001 complete [11951.498454] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [11954.198881] Key type lgssc unregistered [11954.608814] LNet: 339608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11954.618534] LNetError: 339608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11954.631977] LNet: Removed LNI 192.168.202.115@tcp [11955.647088] Key type .llcrypt unregistered [11955.651703] Key type ._llcrypt unregistered [11976.594297] Key type ._llcrypt registered [11976.596408] Key type .llcrypt registered [11976.738251] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [11994.541298] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [11996.051040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11996.158127] alg: No test for adler32 (adler32-zlib) [11997.453121] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11997.720487] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [11999.420138] Key type lgssc registered [12000.962283] Lustre: Echo OBD driver; http://www.lustre.org/ [12044.715413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12044.816855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12046.280933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12046.346855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12046.501648] Lustre: lustre-MDT0000: new disk, initializing [12046.612376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12046.641739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12051.830847] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12063.131712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12063.289898] Lustre: 342772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12063.348220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12063.353758] Lustre: Skipped 1 previous similar message [12063.462789] Lustre: lustre-MDT0001: new disk, initializing [12063.576952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12063.615407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12063.630703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12069.262747] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12081.114580] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12088.071467] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12096.516366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12096.950289] Lustre: lustre-OST0000: new disk, initializing [12096.956053] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12096.973099] Lustre: 344741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12097.142942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12104.686948] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12105.299090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12105.319859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12105.486769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12116.101088] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12122.201724] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12122.533859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12127.990735] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12128.381850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12131.296448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12131.306375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12131.316241] LustreError: Skipped 1 previous similar message [12131.317418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12131.342821] Lustre: Skipped 1 previous similar message [12136.421617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12136.430438] Lustre: Skipped 2 previous similar messages [12141.539343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12141.561365] Lustre: Skipped 1 previous similar message [12145.119152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12145.270635] Lustre: server umount lustre-OST0000 complete [12150.763090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12150.774134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12150.795948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12151.783551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12155.709156] Lustre: server umount lustre-MDT0000 complete [12160.318968] LustreError: 342765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786686064 with bad export cookie 8920346400122098428 [12160.327591] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12160.334729] LustreError: 342765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12160.781733] Lustre: server umount lustre-MDT0001 complete [12171.177550] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:41:13 (1786686073) [12189.507894] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [12200.801691] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [12236.479087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12236.705543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12236.729715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12236.814593] Lustre: lustre-MDT0000: new disk, initializing [12236.915989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12236.965444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12242.179549] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12253.694381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12253.923667] Lustre: 349508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12254.067360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12254.071921] Lustre: Skipped 1 previous similar message [12254.173825] Lustre: lustre-MDT0001: new disk, initializing [12254.368739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12254.425563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12254.450617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12260.313737] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12270.843590] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12276.861534] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12284.147842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12284.475652] Lustre: lustre-OST0000: new disk, initializing [12284.480078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12284.488754] Lustre: 351475:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12284.581987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12291.138873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12291.151170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12291.278183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12291.373756] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12301.930817] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12337.633181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12337.640768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12337.650478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12337.653090] Lustre: Skipped 2 previous similar messages [12342.767402] LustreError: 351459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12342.788733] LustreError: 351459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12342.908503] Lustre: server umount lustre-OST0000 complete [12347.898072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12347.911900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12347.928435] Lustre: Skipped 2 previous similar messages [12347.953606] Lustre: Skipped 1 previous similar message [12353.621097] Lustre: server umount lustre-MDT0000 complete [12358.021243] LustreError: 349498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786686262 with bad export cookie 8920346400122099310 [12358.031205] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12358.038409] LustreError: 349498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12358.638535] Lustre: server umount lustre-MDT0001 complete [12370.806641] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12373.373468] Key type lgssc unregistered [12373.601742] LNet: 353656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12373.607762] LNetError: 353656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12374.630912] LNet: Removed LNI 192.168.202.115@tcp [12375.516751] Key type .llcrypt unregistered [12375.517989] Key type ._llcrypt unregistered [12483.732852] Key type ._llcrypt registered [12483.736776] Key type .llcrypt registered [12483.853477] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12484.368498] Key type .llcrypt unregistered [12484.369746] Key type ._llcrypt unregistered [12569.185549] Key type ._llcrypt registered [12569.189169] Key type .llcrypt registered [12569.276497] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12569.568374] Key type .llcrypt unregistered [12569.572661] Key type ._llcrypt unregistered [12600.653772] Key type ._llcrypt registered [12600.657558] Key type .llcrypt registered [12600.762981] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12601.078943] Key type .llcrypt unregistered [12601.084054] Key type ._llcrypt unregistered [12631.304687] Key type ._llcrypt registered [12631.308939] Key type .llcrypt registered [12631.477917] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12631.833277] Key type .llcrypt unregistered [12631.835520] Key type ._llcrypt unregistered [12706.653799] Key type ._llcrypt registered [12706.655258] Key type .llcrypt registered [12706.750449] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12706.989563] Key type .llcrypt unregistered [12706.991484] Key type ._llcrypt unregistered [12735.169455] Key type ._llcrypt registered [12735.172184] Key type .llcrypt registered [12735.302783] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12735.654444] Key type .llcrypt unregistered [12735.657381] Key type ._llcrypt unregistered [12764.860385] Key type ._llcrypt registered [12764.867414] Key type .llcrypt registered [12764.998796] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12765.463406] Key type .llcrypt unregistered [12765.471417] Key type ._llcrypt unregistered [12797.889554] Key type ._llcrypt registered [12797.891277] Key type .llcrypt registered [12798.042142] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12798.451541] Key type .llcrypt unregistered [12798.463273] Key type ._llcrypt unregistered [12835.058535] Key type ._llcrypt registered [12835.062466] Key type .llcrypt registered [12835.200718] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12835.510690] Key type .llcrypt unregistered [12835.520833] Key type ._llcrypt unregistered [12865.632656] Key type ._llcrypt registered [12865.633802] Key type .llcrypt registered [12865.726972] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12866.126341] Key type .llcrypt unregistered [12866.129142] Key type ._llcrypt unregistered [12917.624339] Key type ._llcrypt registered [12917.626309] Key type .llcrypt registered [12917.715824] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12918.045175] Key type .llcrypt unregistered [12918.047391] Key type ._llcrypt unregistered [12946.378374] Key type ._llcrypt registered [12946.380637] Key type .llcrypt registered [12946.471914] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12946.739204] Key type .llcrypt unregistered [12946.745596] Key type ._llcrypt unregistered [12976.708561] Key type ._llcrypt registered [12976.710616] Key type .llcrypt registered [12976.817310] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12977.182293] Key type .llcrypt unregistered [12977.189100] Key type ._llcrypt unregistered [13031.484245] Key type ._llcrypt registered [13031.487310] Key type .llcrypt registered [13031.598333] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [13046.216984] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [13047.778217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13047.830287] alg: No test for adler32 (adler32-zlib) [13048.904546] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [13049.159744] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [13050.879222] Key type lgssc registered [13052.079109] Lustre: Echo OBD driver; http://www.lustre.org/ [13087.750514] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13087.802683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13089.156731] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13089.180288] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13089.279252] Lustre: tst250fs-MDT0000: new disk, initializing [13089.458810] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13089.489924] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13094.745932] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [13106.867589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13107.133904] Lustre: 373985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13107.268482] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13107.277070] Lustre: Skipped 1 previous similar message [13107.344315] Lustre: tst250fs-MDT0001: new disk, initializing [13107.479059] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13107.553489] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13107.567324] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13113.253367] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [13118.111744] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13124.801148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13125.090645] Lustre: tst250fs-OST0000: new disk, initializing [13125.094645] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13125.104396] Lustre: 375620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13125.197640] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13130.867977] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13130.893818] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13131.059070] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13131.848242] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [13142.801711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13142.979539] Lustre: tst250fs-OST0001: new disk, initializing [13142.982535] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13142.987614] Lustre: 376490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13143.068963] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13149.943713] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [13150.858914] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13150.871986] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13151.033666] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13156.063475] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13158.610243] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13158.614843] Lustre: Skipped 2 previous similar messages [13179.365658] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13179.380057] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13179.402371] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13181.922218] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13181.939924] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13185.245411] Lustre: server umount tst250fs-MDT0000 complete [13189.298798] LustreError: 375619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786687093 with bad export cookie 2014520796726773462 [13189.303815] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13189.310428] LustreError: 375619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13189.747373] Lustre: server umount tst250fs-MDT0001 complete [13203.299616] Lustre: server umount tst250fs-OST0000 complete [13217.972481] Lustre: server umount tst250fs-OST0001 complete [13238.250132] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [13246.248415] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [13279.384856] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:59:42 (1786687182) [13280.620588] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13297.366864] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [13309.476396] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [13346.454969] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13068 sec ========= 02:00:49 (1786687249) [13348.245276] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:00:51 (1786687251) === [13351.279483] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:00:54 (1786687254) === [13391.885352] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13395.031319] Key type lgssc unregistered [13395.477831] LNet: 385651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13395.486934] LNetError: 385651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13395.510617] LNet: Removed LNI 192.168.202.115@tcp [13396.399318] Key type .llcrypt unregistered [13396.401953] Key type ._llcrypt unregistered