[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509360290 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002498] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004000] kvm-guest: setup PV IPIs [ 0.005000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006012] pid_max: default: 32768 minimum: 301 [ 0.008056] LSM: Security Framework initializing [ 0.010060] Yama: becoming mindful. [ 0.011062] SELinux: Initializing. [ 0.012074] *** VALIDATE selinux *** [ 0.020784] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025723] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028000] *** VALIDATE tmpfs *** [ 0.029320] *** VALIDATE proc *** [ 0.031033] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034129] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036027] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.039889] debug: unmapping init [mem 0xffffffffab859000-0xffffffffab860fff] [ 0.041843] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043022] ... version: 2 [ 0.044008] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.046011] ... value mask: 0000ffffffffffff [ 0.047014] ... max period: 00007fffffffffff [ 0.048011] ... fixed-purpose events: 3 [ 0.049014] ... event mask: 000000070000000f [ 0.050278] rcu: Hierarchical SRCU implementation. [ 0.052429] smp: Bringing up secondary CPUs ... [ 0.053629] x86: Booting SMP configuration: [ 0.054029] .... node #0, CPUs: #1 #2 #3 [ 0.059350] smp: Brought up 1 node, 4 CPUs [ 0.061023] smpboot: Max logical packages: 1 [ 0.062017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.189021] node 0 deferred pages initialised in 121ms [ 0.195017] devtmpfs: initialized [ 0.196300] x86/mm: Memory block size: 128MB [ 0.200642] gcov: version magic: 0x41383552 [ 0.204426] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.205130] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.206348] pinctrl core: initialized pinctrl subsystem [ 0.207253] [ 0.207958] ************************************************************* [ 0.208019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209015] ** ** [ 0.210015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.211014] ** ** [ 0.212024] ** This means that this kernel is built to expose internal ** [ 0.213020] ** IOMMU data structures, which may compromise security on ** [ 0.214022] ** your system. ** [ 0.215019] ** ** [ 0.216013] ** If you see this message and you are not debugging the ** [ 0.217016] ** kernel, report this immediately to your vendor! ** [ 0.218020] ** ** [ 0.219020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.220018] ************************************************************* [ 0.222000] NET: Registered protocol family 16 [ 0.222000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.226146] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.228152] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.234335] cpuidle: using governor menu [ 0.236000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.239960] PCI: Using configuration type 1 for base access [ 0.242173] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.252446] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.253031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.256066] cryptd: max_cpu_qlen set to 1000 [ 0.265052] ACPI: Added _OSI(Module Device) [ 0.267041] ACPI: Added _OSI(Processor Device) [ 0.268000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.269036] ACPI: Added _OSI(Processor Aggregator Device) [ 0.278778] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.297000] ACPI: Interpreter enabled [ 0.300085] ACPI: PM: (supports S0 S3 S4 S5) [ 0.301015] ACPI: Using IOAPIC for interrupt routing [ 0.303153] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.308490] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.337160] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.340060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.344028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.349265] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.361000] acpiphp: Slot [2] registered [ 0.363206] acpiphp: Slot [3] registered [ 0.364145] acpiphp: Slot [4] registered [ 0.366168] acpiphp: Slot [5] registered [ 0.367174] acpiphp: Slot [6] registered [ 0.368167] acpiphp: Slot [7] registered [ 0.370140] acpiphp: Slot [8] registered [ 0.371152] acpiphp: Slot [9] registered [ 0.372156] acpiphp: Slot [10] registered [ 0.373186] acpiphp: Slot [11] registered [ 0.375121] acpiphp: Slot [12] registered [ 0.376092] acpiphp: Slot [13] registered [ 0.377104] acpiphp: Slot [14] registered [ 0.378000] acpiphp: Slot [15] registered [ 0.379146] acpiphp: Slot [16] registered [ 0.381122] acpiphp: Slot [17] registered [ 0.392129] acpiphp: Slot [18] registered [ 0.393000] acpiphp: Slot [19] registered [ 0.393196] acpiphp: Slot [20] registered [ 0.397388] acpiphp: Slot [21] registered [ 0.400583] acpiphp: Slot [22] registered [ 0.404328] acpiphp: Slot [23] registered [ 0.407578] acpiphp: Slot [24] registered [ 0.411317] acpiphp: Slot [25] registered [ 0.414578] acpiphp: Slot [26] registered [ 0.418137] acpiphp: Slot [27] registered [ 0.424114] acpiphp: Slot [28] registered [ 0.426095] acpiphp: Slot [29] registered [ 0.432178] acpiphp: Slot [30] registered [ 0.434089] acpiphp: Slot [31] registered [ 0.439088] PCI host bridge to bus 0000:00 [ 0.440020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.442021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.444038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.445000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.451478] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.459060] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.463391] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.470000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.477371] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.492017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.501842] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.505056] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.507022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.509025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.515581] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.519612] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.524048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.525593] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.530016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.541000] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.546021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.553295] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.558018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.568024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.583028] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.592960] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.600018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.604023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.633024] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.649799] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.658019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.669020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.692025] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.708021] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.720023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.728024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.738023] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.743437] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.747000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.749016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.761049] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.769000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.778030] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.783019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.795021] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.804408] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.807488] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.809694] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.813436] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.817277] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.821225] iommu: Default domain type: Passthrough [ 0.824411] SCSI subsystem initialized [ 0.826160] ACPI: bus type USB registered [ 0.827156] usbcore: registered new interface driver usbfs [ 0.829101] usbcore: registered new interface driver hub [ 0.831092] usbcore: registered new device driver usb [ 0.832292] pps_core: LinuxPPS API ver. 1 registered [ 0.835018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.841081] PTP clock support registered [ 0.843220] EDAC MC: Ver: 3.0.0 [ 0.845170] PCI: Using ACPI for IRQ routing [ 0.848038] NetLabel: Initializing [ 0.849000] NetLabel: domain hash size = 128 [ 0.851017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.853109] NetLabel: unlabeled traffic allowed by default [ 0.857441] vgaarb: loaded [ 0.860450] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.861016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.870194] clocksource: Switched to clocksource kvm-clock [ 1.083187] VFS: Disk quotas dquot_6.6.0 [ 1.085221] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.087821] *** VALIDATE ramfs *** [ 1.089529] *** VALIDATE hugetlbfs *** [ 1.091236] pnp: PnP ACPI init [ 1.094674] pnp: PnP ACPI: found 6 devices [ 1.118145] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.121451] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.123469] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.126023] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.128824] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.131329] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.134158] NET: Registered protocol family 2 [ 1.136689] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.141730] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.144944] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.152130] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.154902] TCP: Hash tables configured (established 65536 bind 65536) [ 1.157144] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.159550] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.161722] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.164266] NET: Registered protocol family 1 [ 1.168261] RPC: Registered named UNIX socket transport module. [ 1.170781] RPC: Registered udp transport module. [ 1.172108] RPC: Registered tcp transport module. [ 1.173391] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.175106] NET: Registered protocol family 44 [ 1.177240] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.180844] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.182750] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.187266] PCI: CLS 0 bytes, default 64 [ 1.189725] Unpacking initramfs... [ 2.953464] debug: unmapping init [mem 0xffff9d5c3cc54000-0xffff9d5c3ffbffff] [ 2.958089] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.960300] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.963261] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.762380] Initialise system trusted keyrings [ 3.764098] Key type blacklist registered [ 3.766321] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.775420] zbud: loaded [ 3.778745] *** VALIDATE nfs *** [ 3.779800] *** VALIDATE nfs4 *** [ 3.781155] pstore: using deflate compression [ 3.785194] Platform Keyring initialized [ 3.974022] NET: Registered protocol family 38 [ 3.975858] Key type asymmetric registered [ 3.977179] Asymmetric key parser 'x509' registered [ 3.990273] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.997021] io scheduler mq-deadline registered [ 3.998296] io scheduler kyber registered [ 3.999605] io scheduler bfq registered [ 4.008911] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.015112] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.036625] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.039836] ACPI: Power Button [PWRF] [ 4.355134] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.651703] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.032072] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.163977] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.351449] hrtimer: interrupt took 7650498 ns [ 5.595953] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.657077] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.716227] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.725517] Non-volatile memory driver v1.3 [ 5.726956] Linux agpgart interface v0.103 [ 5.805270] virtio_blk virtio1: [vda] 133248 512-byte logical blocks (68.2 MB/65.1 MiB) [ 5.808265] vda: detected capacity change from 0 to 68222976 [ 5.850328] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.858302] vdb: detected capacity change from 0 to 1073741824 [ 5.891241] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.899405] vdc: detected capacity change from 0 to 2621440000 [ 5.939690] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.946452] vdd: detected capacity change from 0 to 2621440000 [ 5.981938] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.988165] vde: detected capacity change from 0 to 4294967296 [ 6.024522] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.035781] vdf: detected capacity change from 0 to 4294967296 [ 6.052409] libphy: Fixed MDIO Bus: probed [ 6.063367] usbcore: registered new interface driver usbserial_generic [ 6.068396] usbserial: USB Serial support registered for generic [ 6.073699] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.079804] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.081783] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.090541] mousedev: PS/2 mouse device common for all mice [ 6.097385] rtc_cmos 00:05: RTC can wake from S4 [ 6.105982] rtc_cmos 00:05: registered as rtc0 [ 6.107519] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.111485] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.111544] intel_pstate: CPU model not supported [ 6.125954] hid: raw HID events driver (C) Jiri Kosina [ 6.126130] usbcore: registered new interface driver usbhid [ 6.126133] usbhid: USB HID core driver [ 6.126232] drop_monitor: Initializing network drop monitor service [ 6.126355] Initializing XFRM netlink socket [ 6.126672] NET: Registered protocol family 10 [ 6.130437] Segment Routing with IPv6 [ 6.152217] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.154449] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.192170] NET: Registered protocol family 17 [ 6.204842] mpls_gso: MPLS GSO support [ 6.211522] RAS: Correctable Errors collector initialized. [ 6.217637] AVX version of gcm_enc/dec engaged. [ 6.221569] AES CTR mode by8 optimization enabled [ 6.395512] sched_clock: Marking stable (6395496163, 0)->(7527009757, -1131513594) [ 6.399274] registered taskstats version 1 [ 6.401837] Loading compiled-in X.509 certificates [ 6.403567] zswap: loaded using pool lzo/zbud [ 6.462198] Key type big_key registered [ 6.501278] Key type encrypted registered [ 6.502881] ima: No TPM chip found, activating TPM-bypass! [ 6.505841] ima: Allocated hash algorithm: sha1 [ 6.507858] ima: No architecture policies found [ 6.516206] evm: Initialising EVM extended attributes: [ 6.518133] evm: security.selinux [ 6.519297] evm: security.ima [ 6.520403] evm: security.capability [ 6.521651] evm: HMAC attrs: 0x1 [ 6.525199] rtc_cmos 00:05: setting system clock to 2025-10-10 19:42:30 UTC (1760125350) [ 6.538526] debug: unmapping init [mem 0xffffffffac803000-0xffffffffac9fffff] [ 6.543360] debug: unmapping init [mem 0xffffffffab582000-0xffffffffab858fff] [ 6.555490] Write protecting the kernel read-only data: 28672k [ 6.566172] debug: unmapping init [mem 0xffffffffa9c03000-0xffffffffa9dfffff] [ 6.579375] debug: unmapping init [mem 0xffffffffaa514000-0xffffffffaa5fffff] [ 6.653606] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.670403] systemd[1]: Detected virtualization kvm. [ 6.672471] systemd[1]: Detected architecture x86-64. [ 6.680569] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.762674] systemd[1]: No hostname configured. [ 6.763970] systemd[1]: Set hostname to . [ 6.765465] random: systemd: uninitialized urandom read (16 bytes read) [ 6.767303] systemd[1]: Initializing machine ID from random generator. [ 7.021177] random: systemd: uninitialized urandom read (16 bytes read) [ 7.023446] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 7.027216] random: systemd: uninitialized urandom read (16 bytes read) [ 7.029921] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 7.035395] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.553369] device-mapper: uevent: version 1.0.3 [ 8.557431] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.295482] random: fast init done [ 10.313215] virtio_net virtio0 ens2: renamed from eth0 [ 10.523196] scsi host0: ata_piix [ 10.713179] scsi host1: ata_piix [ 10.714371] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.716962] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.048653] random: crng init done [ 16.049863] random: 7 urandom warning(s) missed due to ratelimiting [ 18.650253] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.769283] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.512088] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.200313] SELinux: Disabled at runtime. [ 24.319542] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.329655] systemd[1]: Detected virtualization kvm. [ 24.334474] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.590616] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.597657] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.615596] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.619472] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.622424] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.669306] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.716283] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl s[ 26.423936] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS tatus systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 26.959615] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.041640] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.119332] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.418089] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.463792] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 33.600310] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 34.822379] NFS: Registering the id_resolver key type [ 34.824337] Key type id_resolver registered [ 34.830205] Key type id_legacy registered [ ***] A start job is running for Configur…-only root support (9s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg219-server login: [ 84.013722] libcfs: loading out-of-tree module taints kernel. [ 84.032991] Key type ._llcrypt registered [ 84.034908] Key type .llcrypt registered [ 84.128781] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 95.108676] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 96.033349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.041890] alg: No test for adler32 (adler32-zlib) [ 97.178055] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 97.637086] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 99.288147] Key type lgssc registered [ 100.291292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 111.605310] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 113.165877] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.142498] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.309311] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.932092] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 150.785983] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 161.783162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.836821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.855708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.093774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.125694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.192330] Lustre: lustre-MDT0000: new disk, initializing [ 163.289832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.314935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 166.200028] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.942437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.003353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 176.066402] Lustre: 6497:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 176.090469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 176.094758] Lustre: Skipped 1 previous similar message [ 176.165387] Lustre: lustre-MDT0001: new disk, initializing [ 176.209469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.235526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 176.243277] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.633364] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.515033] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.864985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.910232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 190.052881] Lustre: lustre-OST0000: new disk, initializing [ 190.057187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.094650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.693522] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.117821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.137894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.218390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.920824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.975719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 206.050681] Lustre: lustre-OST0001: new disk, initializing [ 206.057352] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 206.125069] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.521345] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.508662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.511944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.572290] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 221.528307] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.518205] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 242.595409] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.190344] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing yml_node [ 250.600153] Lustre: DEBUG MARKER: Client: 2.16.58.106 [ 252.785989] Lustre: DEBUG MARKER: MDS: 2.16.58.106 [ 254.966415] Lustre: DEBUG MARKER: OSS: 2.16.58.106 [ 256.119049] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 10 15:46:39 EDT 2025 [ 271.184332] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 272.343354] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.737644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 288.742444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.757529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.272806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.273213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.277607] Lustre: Skipped 1 previous similar message [ 290.294130] Lustre: Skipped 1 previous similar message [ 292.636616] LustreError: 12200:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 294.695957] LustreError: 12200:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 294.698942] LustreError: 12200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 294.773918] Lustre: server umount lustre-MDT0000 complete [ 297.610056] LustreError: 6490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760125641 with bad export cookie 16572844273833345508 [ 297.616838] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.624483] LustreError: 6490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 300.513325] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.515343] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.515611] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.515615] Lustre: Skipped 2 previous similar messages [ 300.521469] Lustre: Skipped 2 previous similar messages [ 300.546719] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 302.561441] LustreError: 6504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.562823] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.580814] Lustre: Skipped 2 previous similar messages [ 303.835472] LustreError: 12403:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 303.839167] LustreError: 12403:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 304.082318] Lustre: server umount lustre-MDT0001 complete [ 307.226152] LustreError: 12604:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 307.229576] LustreError: 12604:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 307.271271] Lustre: server umount lustre-OST0000 complete [ 310.122167] Lustre: server umount lustre-OST0001 complete [ 314.767785] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 321.525538] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 328.120485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.416030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 336.733176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.998547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 346.219322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 346.268771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.427945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 346.456920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 346.526620] Lustre: lustre-MDT0000: new disk, initializing [ 346.602976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 346.616675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 350.163566] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 358.246778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 358.305967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 358.374659] Lustre: 14987:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 358.427791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 358.431285] Lustre: Skipped 1 previous similar message [ 358.482688] Lustre: lustre-MDT0001: new disk, initializing [ 358.561323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 358.594787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 358.605940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 361.612431] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 367.525758] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 370.336693] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 375.350779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 375.397843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 375.583859] Lustre: lustre-OST0000: new disk, initializing [ 375.587257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 375.660873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.562744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 377.568151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 377.671288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 379.237911] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 384.355436] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 386.766476] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 386.913724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 389.544787] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 389.700669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 393.189940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 393.200119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 393.200740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 393.224820] LustreError: Skipped 1 previous similar message [ 396.992171] LustreError: 17923:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 397.002955] LustreError: 17923:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 397.131663] Lustre: server umount lustre-OST0000 complete [ 399.843988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 399.849316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.855959] Lustre: Skipped 1 previous similar message [ 403.431188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 403.437276] Lustre: Skipped 3 previous similar messages [ 414.176835] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 414.236047] LustreError: 18124:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 414.238032] LustreError: 18124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 414.328810] Lustre: server umount lustre-MDT0000 complete [ 416.656103] LustreError: 14981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760125760 with bad export cookie 16572844273833347041 [ 416.657033] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 416.663136] LustreError: 14981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 416.984474] Lustre: server umount lustre-MDT0001 complete [ 419.341566] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:49:22 (1760125762) [ 420.731229] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 421.805506] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:49:25 (1760125765) [ 425.479069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.713868] LustreError: 18999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 425.727501] LustreError: 18999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 425.761482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.863956] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 431.075754] LustreError: 18999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.772025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 435.150960] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 438.576902] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 440.895776] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 445.128055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 448.931912] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 452.185540] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 460.770731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 460.775867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.781610] Lustre: Skipped 1 previous similar message [ 460.784502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 460.787672] Lustre: Skipped 4 previous similar messages [ 466.036495] LustreError: 21242:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 466.039861] LustreError: 21242:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 466.147524] Lustre: server umount lustre-OST0000 complete [ 468.963436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 468.969143] LustreError: Skipped 1 previous similar message [ 474.391533] Lustre: server umount lustre-MDT0000 complete [ 476.765635] LustreError: 18980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760125820 with bad export cookie 16572844273833347923 [ 476.767470] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 476.772967] LustreError: 18980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 477.011479] Lustre: server umount lustre-MDT0001 complete [ 484.537429] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 486.128718] Key type lgssc unregistered [ 486.332347] LNet: 22180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 486.336702] LNetError: 22180:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 486.345473] LNet: Removed LNI 192.168.202.119@tcp [ 486.869282] Key type .llcrypt unregistered [ 486.871198] Key type ._llcrypt unregistered [ 499.405149] Key type ._llcrypt registered [ 499.407714] Key type .llcrypt registered [ 499.471494] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 500.021986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 500.043256] alg: No test for adler32 (adler32-zlib) [ 500.979692] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 501.140491] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 502.768217] Key type lgssc registered [ 503.377947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 508.480355] loop: module loaded [ 508.511981] loop0: detected capacity change from 0 to 204800000 [ 508.604186] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 508.612556] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 508.781039] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 508.787100] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 508.826720] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 508.854768] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 508.884123] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 508.943089] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 512.346104] loop0: detected capacity change from 0 to 204800000 [ 512.463979] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 512.468830] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 512.474444] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 512.566077] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 515.689930] loop0: detected capacity change from 0 to 204800000 [ 519.100743] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 519.146913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 519.155113] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.310869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 520.360353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 520.417845] Lustre: lustre-MDT0000: new disk, initializing [ 520.494395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.511449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 522.549916] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 527.681087] loop1: detected capacity change from 0 to 204800000 [ 530.953503] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 531.009719] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 531.183126] Lustre: lustre-OST07c6: new disk, initializing [ 531.187485] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 531.193306] Lustre: Skipped 1 previous similar message [ 531.247461] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 534.618394] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 535.266658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 535.271362] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 535.295143] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 627.686105] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 627.691710] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.699931] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 632.708285] LustreError: 26503:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 632.823797] Lustre: server umount lustre-OST07c6 complete [ 639.477613] LustreError: 27003:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 639.480498] LustreError: 27003:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 639.651655] Lustre: server umount lustre-MDT0000 complete [ 650.174407] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 652.119867] Key type lgssc unregistered [ 652.330499] LNet: 27837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.336500] LNetError: 27837:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.351451] LNet: Removed LNI 192.168.202.119@tcp [ 653.896686] Key type .llcrypt unregistered [ 653.898503] Key type ._llcrypt unregistered [ 664.145083] Key type ._llcrypt registered [ 664.146284] Key type .llcrypt registered [ 664.209247] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 664.691527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 664.700611] alg: No test for adler32 (adler32-zlib) [ 665.666433] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 665.836673] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 667.504175] Key type lgssc registered [ 668.182408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 673.474736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 673.500036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.793707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.709991] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 683.690471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 683.916356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 686.338820] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 689.969814] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 692.205520] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 696.850649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 697.112667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.183280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 700.963447] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 704.370475] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 711.798900] LustreError: 31279:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 717.796483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 717.797527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.799505] LustreError: Skipped 1 previous similar message [ 717.810469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 718.819140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.819757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 718.820235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 718.835606] Lustre: Skipped 1 previous similar message [ 723.195097] LustreError: 31529:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 723.275614] Lustre: server umount lustre-OST0000 complete [ 729.063307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.070050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 729.072758] Lustre: Skipped 1 previous similar message [ 731.525794] LustreError: 31730:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 731.531824] LustreError: 31730:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 731.581863] Lustre: server umount lustre-MDT0000 complete [ 733.594477] LustreError: 29066:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126077 with bad export cookie 12349549064604437572 [ 733.597988] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 733.599450] LustreError: 29066:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 733.682886] LustreError: 31934:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 733.685950] LustreError: 31934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 733.823251] Lustre: server umount lustre-MDT0001 complete [ 739.749642] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 741.847795] Key type lgssc unregistered [ 742.078839] LNet: 32466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.083465] LNetError: 32466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.095256] LNet: Removed LNI 192.168.202.119@tcp [ 742.748440] Key type .llcrypt unregistered [ 742.749804] Key type ._llcrypt unregistered [ 755.435162] Key type ._llcrypt registered [ 755.437126] Key type .llcrypt registered [ 755.500639] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 756.252658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 756.274084] alg: No test for adler32 (adler32-zlib) [ 757.228349] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 757.407585] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 759.040178] Key type lgssc registered [ 759.781746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 764.539594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 764.559612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 765.891851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.246258] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 773.840627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 774.079434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 776.753099] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 780.403432] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 782.996720] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 787.479133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 787.755718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.833043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 791.227815] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 794.787821] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 804.323555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 804.323738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 804.327892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 804.329049] Lustre: Skipped 1 previous similar message [ 809.444975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 809.448363] Lustre: Skipped 2 previous similar messages [ 810.287369] LustreError: 35937:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 810.381646] Lustre: server umount lustre-OST0000 complete [ 814.565503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.572958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 814.576739] Lustre: Skipped 1 previous similar message [ 814.583251] Lustre: Skipped 1 previous similar message [ 818.519740] LustreError: 36139:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 818.527148] LustreError: 36139:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 818.606607] Lustre: server umount lustre-MDT0000 complete [ 819.682298] LustreError: 34481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 819.690079] LustreError: 34481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 820.688638] LustreError: 33674:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126164 with bad export cookie 512551619569557667 [ 820.692907] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.694953] LustreError: 33674:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 820.779446] LustreError: 36341:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 820.782057] LustreError: 36341:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 820.888168] Lustre: server umount lustre-MDT0001 complete [ 825.675678] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 827.234375] Key type lgssc unregistered [ 827.415327] LNet: 36873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.423244] LNetError: 36873:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.444447] LNet: Removed LNI 192.168.202.119@tcp [ 827.938979] Key type .llcrypt unregistered [ 827.940461] Key type ._llcrypt unregistered [ 840.831204] Key type ._llcrypt registered [ 840.832743] Key type .llcrypt registered [ 840.885400] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 841.411337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 841.420047] alg: No test for adler32 (adler32-zlib) [ 842.299731] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 842.406888] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 844.008201] Key type lgssc registered [ 844.537867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 848.511623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 848.533091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.759608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 851.466092] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 855.478516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 855.654427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.324674] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 859.630859] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 861.173744] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 864.177328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 864.339059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.950116] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 869.375556] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 874.605644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 880.100237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.102643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 880.108464] Lustre: Skipped 1 previous similar message [ 880.115983] Lustre: Skipped 2 previous similar messages [ 882.097931] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 885.217701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 885.221629] Lustre: Skipped 2 previous similar messages [ 885.653637] LustreError: 40493:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 885.703739] Lustre: server umount lustre-MDT0000 complete [ 887.204158] LustreError: 38248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.212021] LustreError: 38248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 887.269442] LustreError: 38233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126231 with bad export cookie 6711773208458205681 [ 887.272732] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 887.273591] LustreError: 38233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 890.337963] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.338554] LustreError: 38246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 890.344471] Lustre: Skipped 1 previous similar message [ 890.345468] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 892.327958] LustreError: 39410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.383964] LustreError: 40694:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 893.386984] LustreError: 40694:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 893.490322] Lustre: server umount lustre-MDT0001 complete [ 912.246871] LustreError: 40920:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 912.250501] LustreError: 40920:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 912.351719] Lustre: server umount lustre-OST0000 complete [ 921.950140] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 923.727750] Key type lgssc unregistered [ 923.907928] LNet: 41756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.912480] LNetError: 41756:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.923783] LNet: Removed LNI 192.168.202.119@tcp [ 924.406439] Key type .llcrypt unregistered [ 924.408198] Key type ._llcrypt unregistered [ 937.130825] Key type ._llcrypt registered [ 937.132480] Key type .llcrypt registered [ 937.196599] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 937.604106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 937.660401] alg: No test for adler32 (adler32-zlib) [ 938.572940] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 938.694318] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 940.304161] Key type lgssc registered [ 940.784229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 944.671889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 944.682200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.944252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 948.019373] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 952.400623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 952.621080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 954.441161] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 957.236043] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 958.953202] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 962.656251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 962.893208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.273527] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 969.709456] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 970.224307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 978.401833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 978.404865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.411646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 980.409381] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 980.451018] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 980.452552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.178483] LustreError: 45377:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 984.269623] Lustre: server umount lustre-OST0000 complete [ 1000.930053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.935203] Lustre: Skipped 1 previous similar message [ 1000.935377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.941873] Lustre: Skipped 3 previous similar messages [ 1005.199344] LustreError: 45753:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1005.205624] LustreError: 45753:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1005.252503] Lustre: server umount lustre-MDT0000 complete [ 1006.051635] LustreError: 43855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.063661] LustreError: 43855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1007.382881] LustreError: 43111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126351 with bad export cookie 11387980396130517358 [ 1007.384875] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1007.392705] LustreError: 43111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1013.584401] LustreError: 45955:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1013.588375] LustreError: 45955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1013.750561] Lustre: server umount lustre-MDT0001 complete [ 1018.939872] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1020.653478] Key type lgssc unregistered [ 1020.891605] LNet: 46490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.894492] LNetError: 46490:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.911038] LNet: Removed LNI 192.168.202.119@tcp [ 1021.384449] Key type .llcrypt unregistered [ 1021.385862] Key type ._llcrypt unregistered [ 1034.737469] Key type ._llcrypt registered [ 1034.739106] Key type .llcrypt registered [ 1034.803979] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1035.347675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.355489] alg: No test for adler32 (adler32-zlib) [ 1036.296966] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 1036.434401] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1038.048250] Key type lgssc registered [ 1038.657596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1042.407948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1042.422222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.660356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.618837] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1049.938585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1050.136170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1052.038444] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1054.778192] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1056.828613] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1060.639286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1060.828828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.816607] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1066.956539] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1069.044045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1083.475404] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1088.735469] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1102.059360] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-1016h00m43s [ 1104.370873] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1110.472754] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1111.010230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.014547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.040192] Lustre: Skipped 2 previous similar messages [ 1111.093599] Lustre: Skipped 2 previous similar messages [ 1112.171893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1112.174734] Lustre: Skipped 1 previous similar message [ 1117.631500] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1121.249547] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1121.385730] LustreError: 50451:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1121.580190] Lustre: server umount lustre-MDT0000 complete [ 1123.749303] LustreError: 50021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.779541] LustreError: 50021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1124.834428] LustreError: 47865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.862637] LustreError: 47865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1126.145353] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1129.895045] LustreError: 47860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.712896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.850772] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1134.084900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.147895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1135.087396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1137.347447] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1139.963987] Lustre: DEBUG MARKER: Wait for df (36058) ... [ 1141.196621] Lustre: DEBUG MARKER: done [ 1144.298143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1144.302188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.333159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.339101] Lustre: Skipped 3 previous similar messages [ 1145.322537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.071115] LustreError: 51522:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1150.074821] LustreError: 51522:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1150.200264] Lustre: server umount lustre-OST0000 complete [ 1155.557394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1155.560293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.568720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.572017] Lustre: Skipped 2 previous similar messages [ 1167.328814] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1167.421136] LustreError: 51723:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1167.427590] LustreError: 51723:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1167.489970] Lustre: server umount lustre-MDT0000 complete [ 1170.182837] LustreError: 47846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126514 with bad export cookie 8459212935993757911 [ 1170.185799] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.193102] LustreError: 47846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1170.346274] LustreError: 51926:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1170.349220] LustreError: 51926:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1170.673197] Lustre: server umount lustre-MDT0001 complete [ 1179.265152] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1181.236940] Key type lgssc unregistered [ 1181.442451] LNet: 52458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.451756] LNetError: 52458:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1181.462516] LNet: Removed LNI 192.168.202.119@tcp [ 1182.067267] Key type .llcrypt unregistered [ 1182.068733] Key type ._llcrypt unregistered [ 1206.618242] Key type ._llcrypt registered [ 1206.620023] Key type .llcrypt registered [ 1206.720946] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 1218.258620] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1219.085644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1219.098033] alg: No test for adler32 (adler32-zlib) [ 1220.080483] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 1220.318556] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1221.968177] Key type lgssc registered [ 1223.153241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1230.135380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1234.944706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1239.802791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1244.271681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1249.949744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1249.986733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1249.997730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.192268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1251.213951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1251.260522] Lustre: lustre-MDT0000: new disk, initializing [ 1251.339968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.362720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1254.589668] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1263.490655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1263.555396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1263.663807] Lustre: 56387:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1263.768659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1263.771727] Lustre: Skipped 1 previous similar message [ 1263.877881] Lustre: lustre-MDT0001: new disk, initializing [ 1263.977930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1264.017716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1264.040387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1267.181588] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1272.318054] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1274.787319] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1280.343753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1280.441940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1280.693539] Lustre: lustre-OST0000: new disk, initializing [ 1280.696570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1280.785400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.568898] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1290.286749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1290.295300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1290.369538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1291.876974] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1295.164244] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1295.335497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1298.451212] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1298.615483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1300.448988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.458866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.470567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1300.966107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.968876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.987070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1306.083989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1306.086790] Lustre: Skipped 1 previous similar message [ 1311.209106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1311.212273] Lustre: Skipped 1 previous similar message [ 1314.272127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1314.313154] LustreError: 59324:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1314.440778] Lustre: server umount lustre-OST0000 complete [ 1320.419139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1320.424270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1320.441120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1323.749569] LustreError: 59528:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1323.752864] LustreError: 59528:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1323.821427] Lustre: server umount lustre-MDT0000 complete [ 1326.664854] LustreError: 56378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126670 with bad export cookie 14658490297125080350 [ 1326.675754] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1326.681975] LustreError: 56378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1326.809923] LustreError: 59730:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1326.813435] LustreError: 59730:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1326.995418] Lustre: server umount lustre-MDT0001 complete [ 1333.487793] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 16:04:36 (1760126676) [ 1339.142035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.486960] LustreError: 60402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.513389] LustreError: 60402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1339.560509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.976601] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1345.000483] LustreError: 60402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.096035] LustreError: 60402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.951504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1350.219548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1354.426128] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1361.784780] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1367.555471] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1377.163958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1377.779714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1384.137715] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1388.989380] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1398.359312] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-1016h05m41s [ 1399.958101] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1406.117533] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1408.322829] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1408.414233] LustreError: 60398:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1428.969206] Lustre: lustre-MDT0000: Client 24720b13-d9b4-49a1-9141-f70f19637b04 (at 192.168.202.19@tcp) reconnecting [ 1448.481281] LustreError: 60398:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1450.763881] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1452.836450] Lustre: DEBUG MARKER: done [ 1461.730104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.739896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1461.772551] Lustre: Skipped 2 previous similar messages [ 1461.799415] Lustre: Skipped 3 previous similar messages [ 1463.420132] LustreError: 63405:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1463.423284] LustreError: 63405:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1463.596279] Lustre: server umount lustre-OST0000 complete [ 1466.850730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.867088] LustreError: 63608:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1472.878264] LustreError: 63608:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1472.965655] Lustre: server umount lustre-MDT0000 complete [ 1472.998852] LustreError: 63113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.023465] LustreError: 63113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1476.494284] LustreError: 61121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126820 with bad export cookie 14658490297125081232 [ 1476.498388] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1476.510751] LustreError: 61121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1476.928574] Lustre: server umount lustre-MDT0001 complete [ 1487.465920] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1490.384915] Key type lgssc unregistered [ 1490.632264] LNet: 64340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.644792] LNetError: 64340:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.657516] LNet: Removed LNI 192.168.202.119@tcp [ 1491.410945] Key type .llcrypt unregistered [ 1491.412697] Key type ._llcrypt unregistered [ 1518.590228] Key type ._llcrypt registered [ 1518.591869] Key type .llcrypt registered [ 1518.690653] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 1531.650449] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1532.760766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1532.781977] alg: No test for adler32 (adler32-zlib) [ 1533.844780] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 1534.060410] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1535.696193] Key type lgssc registered [ 1536.569785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1544.649069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1550.840823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1556.646194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1562.202984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1568.333553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1568.371870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1568.384689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.526653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1569.547435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1569.596637] Lustre: lustre-MDT0000: new disk, initializing [ 1569.677061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.691260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1573.195661] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1582.278902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1582.346544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1582.418240] Lustre: 68296:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1582.478379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1582.483878] Lustre: Skipped 1 previous similar message [ 1582.551487] Lustre: lustre-MDT0001: new disk, initializing [ 1582.655944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1582.698707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1582.711142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1586.580520] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1593.227385] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1596.547504] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1602.782166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1602.839835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1603.024231] Lustre: lustre-OST0000: new disk, initializing [ 1603.028702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1603.107181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.312313] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1609.814237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1609.824852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1609.959452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1615.377221] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1618.409626] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1618.607837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1621.861161] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1622.041431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1625.571858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.589121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1625.589635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1625.605922] Lustre: Skipped 1 previous similar message [ 1630.693616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1635.817437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1635.820514] Lustre: Skipped 2 previous similar messages [ 1638.368157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1638.408376] LustreError: 71236:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1638.569561] Lustre: server umount lustre-OST0000 complete [ 1644.005320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1644.012505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.023492] Lustre: Skipped 1 previous similar message [ 1644.027803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1646.055770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.833906] LustreError: 71439:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1647.849653] LustreError: 71439:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1648.060861] Lustre: server umount lustre-MDT0000 complete [ 1651.345413] LustreError: 69079:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126995 with bad export cookie 5857436504998494478 [ 1651.349993] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1651.351773] LustreError: 69079:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1651.474287] LustreError: 71642:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1651.476728] LustreError: 71642:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1651.650802] Lustre: server umount lustre-MDT0001 complete [ 1658.698259] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 16:10:01 (1760127001) [ 1660.137861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1661.402494] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 16:10:04 (1760127004) [ 1670.177636] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1673.778879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.039614] LustreError: 72699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1674.062931] LustreError: 72699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1674.108797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.036145] Lustre: Failing over lustre-MDT0000 [ 1675.042443] LustreError: 72727:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1675.054814] LustreError: 72727:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1675.072584] LustreError: 72727:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1675.215300] LustreError: 72778:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1675.218489] LustreError: 72778:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1675.453760] Lustre: server umount lustre-MDT0000 complete [ 1681.417914] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:10:24 (1760127024) [ 1686.887353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.190936] LustreError: 73348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.237493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.744902] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1692.645509] LustreError: 73348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.772991] LustreError: 73348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.421359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1698.722502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1701.944892] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1706.249064] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1709.104835] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1714.555516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1714.935347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.365627] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1724.067986] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1738.651648] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1744.867402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1744.877083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.886949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1744.889266] Lustre: Skipped 2 previous similar messages [ 1745.693049] LustreError: 75714:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1745.704552] LustreError: 75714:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1745.829420] Lustre: server umount lustre-MDT0000 complete [ 1747.941433] LustreError: 74411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.958530] LustreError: 73326:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760127092 with bad export cookie 5857436504998495640 [ 1748.966637] LustreError: 73326:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1748.966930] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1749.285239] Lustre: server umount lustre-MDT0001 complete [ 1752.335292] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1756.161680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1763.502697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1769.440590] Lustre: 66508:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760127097/real 1760127097] req@00000000f4c6d121 x1845626801298944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760127113 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1769.457177] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.465205] Lustre: Skipped 2 previous similar messages [ 1773.601894] LustreError: 66505:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e54f9858 x1845626801299840/t0(0) o250->MGC192.168.202.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1774.180639] LustreError: 74955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1774.203804] LustreError: 74955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1774.315733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.484623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1775.329020] Lustre: 66509:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760127102/real 1760127102] req@0000000020148dca x1845626801299072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760127118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1778.749643] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1779.361456] Lustre: 66506:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760127107/real 1760127107] req@00000000c3b35d97 x1845626801299328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760127123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1784.800153] Lustre: 66506:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760127112/real 1760127112] req@00000000260b867e x1845626801299584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760127128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1787.727530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1789.225147] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1792.258348] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1797.024778] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1800.542297] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1808.403120] LustreError: 78173:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1808.413200] LustreError: 78173:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1808.549392] Lustre: server umount lustre-MDT0000 complete [ 1808.878988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1808.882805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.890170] LustreError: 74955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1808.899180] LustreError: 74955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1811.765165] LustreError: 73328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760127155 with bad export cookie 5857436504998497544 [ 1811.767451] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1811.774480] LustreError: 73328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1812.162806] Lustre: server umount lustre-MDT0001 complete [ 1817.585789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1825.154101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.305585] Lustre: 66508:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760127158/real 1760127158] req@0000000024fc4437 x1845626801324160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760127174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1830.342286] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.359673] Lustre: Skipped 2 previous similar messages [ 1836.961293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1839.737227] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1840.996198] LustreError: 79228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1841.007317] LustreError: 79228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1843.040262] Lustre: 66509:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760127170/real 1760127170] req@00000000cd63bf3d x1845626801324800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760127186 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1843.053562] Lustre: 66509:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1846.286498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1846.695432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1846.704284] Lustre: Skipped 2 previous similar messages [ 1849.767023] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1849.776793] Lustre: Skipped 1 previous similar message [ 1850.396590] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1855.525870] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1858.247808] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1865.192943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.198323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.207916] Lustre: Skipped 2 previous similar messages [ 1865.221152] Lustre: Skipped 2 previous similar messages [ 1868.864824] LustreError: 80629:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1868.876471] LustreError: 80629:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1869.000302] Lustre: server umount lustre-MDT0000 complete [ 1872.193045] LustreError: 74966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760127216 with bad export cookie 5857436504998498916 [ 1872.195138] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.200252] LustreError: 74966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1872.585589] Lustre: server umount lustre-MDT0001 complete [ 1877.958605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1880.841694] Lustre: DEBUG MARKER: files compared the same [ 1893.135635] Lustre: server umount lustre-OST0000 complete [ 1906.910556] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1909.061839] Key type lgssc unregistered [ 1909.403643] LNet: 82266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.419533] LNetError: 82266:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.439827] LNet: Removed LNI 192.168.202.119@tcp [ 1911.098960] Key type .llcrypt unregistered [ 1911.101652] Key type ._llcrypt unregistered [ 1930.431036] Key type ._llcrypt registered [ 1930.433850] Key type .llcrypt registered [ 1930.606339] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1932.243434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1932.291668] alg: No test for adler32 (adler32-zlib) [ 1933.614909] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 1933.852827] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1935.528163] Key type lgssc registered [ 1936.603828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1942.260496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1942.294493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.694781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.835762] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all [ 1953.255786] LustreError: 83661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.270234] LustreError: 83661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1953.848459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1954.191623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1957.459331] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all [ 1962.997170] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1966.678725] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1972.614537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1972.893726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.521443] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all [ 1984.067353] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1988.271120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1998.822134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1998.824733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.831291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1998.845691] Lustre: Skipped 1 previous similar message [ 2000.942078] LustreError: 85906:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2001.117428] Lustre: server umount lustre-OST0000 complete [ 2005.475141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2005.477817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.497371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.500134] Lustre: Skipped 1 previous similar message [ 2009.061553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.065843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2009.074465] Lustre: Skipped 1 previous similar message [ 2010.588343] LustreError: 86109:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2010.590737] LustreError: 86109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2010.699780] Lustre: server umount lustre-MDT0000 complete [ 2014.022456] LustreError: 83643:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760127357 with bad export cookie 8605734251258762030 [ 2014.023851] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2014.028847] LustreError: 83643:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2014.156856] LustreError: 86310:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2014.160431] LustreError: 86310:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2014.374856] Lustre: server umount lustre-MDT0001 complete [ 2022.776925] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2025.305865] Key type lgssc unregistered [ 2025.578531] LNet: 86841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.593632] LNetError: 86841:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2025.618268] LNet: Removed LNI 192.168.202.119@tcp [ 2026.278662] Key type .llcrypt unregistered [ 2026.280245] Key type ._llcrypt unregistered [ 2044.540848] Key type ._llcrypt registered [ 2044.542956] Key type .llcrypt registered [ 2044.637788] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2045.279495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.395868] alg: No test for adler32 (adler32-zlib) [ 2046.442170] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 2046.624297] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2048.280176] Key type lgssc registered [ 2049.151714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2055.795597] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2055.811180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2071.392875] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2076.512406] LustreError: 88192:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [ 2103.200263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.692447] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2114.020172] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2122.395642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.500289] Lustre: *** cfs_fail_loc=706, val=0*** [ 2122.952805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.032596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2126.553971] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2128.355535] LustreError: 88218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.377164] LustreError: 88218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2133.485977] LustreError: 88218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.260888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2134.593514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2138.342968] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2143.704171] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2146.998690] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2149.858897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2149.867166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.890761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.446026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.454785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.578414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2158.582378] Lustre: Skipped 2 previous similar messages [ 2163.168188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2163.211086] LustreError: 90515:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2163.381504] Lustre: server umount lustre-OST0000 complete [ 2168.801922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.810581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2168.821312] Lustre: Skipped 1 previous similar message [ 2168.830544] Lustre: Skipped 1 previous similar message [ 2173.923197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2181.088139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2181.153874] LustreError: 90719:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2181.157546] LustreError: 90719:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2181.245121] Lustre: server umount lustre-MDT0000 complete [ 2184.165912] LustreError: 89163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.179954] LustreError: 89163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2185.621076] LustreError: 88199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760127529 with bad export cookie 13520089104947914392 [ 2185.637634] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.640592] LustreError: 88199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2185.835574] LustreError: 90924:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2185.837853] LustreError: 90924:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2186.223458] Lustre: server umount lustre-MDT0001 complete [ 2196.963921] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2199.758828] Key type lgssc unregistered [ 2200.038578] LNet: 91455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.050682] LNetError: 91455:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2201.072757] LNet: Removed LNI 192.168.202.119@tcp [ 2201.859564] Key type .llcrypt unregistered [ 2201.861782] Key type ._llcrypt unregistered [ 2221.952329] Key type ._llcrypt registered [ 2221.953755] Key type .llcrypt registered [ 2222.088634] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2223.080731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.117888] alg: No test for adler32 (adler32-zlib) [ 2224.143517] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 2224.400480] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2226.105296] Key type lgssc registered [ 2227.138193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2234.585996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2234.621120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.487337] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2247.791368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2248.662762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2252.570406] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2253.797652] LustreError: 93503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.804282] LustreError: 93503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2258.917142] LustreError: 93507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.939635] LustreError: 93507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2262.166731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2262.660858] LustreError: 93507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.699548] LustreError: 93507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2262.715985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.146043] LustreError: 93503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.580072] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2273.254274] LustreError: 93506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.281883] LustreError: 93506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2279.628330] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.642092] Lustre: MGS: Client fad267ba-bee4-4bad-92e1-901d71d36b90 (at 0@lo) reconnecting [ 2279.879536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.986617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2284.615952] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2300.390274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2300.395991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.408895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2300.416844] Lustre: Skipped 1 previous similar message [ 2304.317166] LustreError: 95469:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2304.613379] Lustre: server umount lustre-OST0000 complete [ 2310.628135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2310.640658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.656759] Lustre: Skipped 1 previous similar message [ 2310.657659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.691016] Lustre: Skipped 2 previous similar messages [ 2315.746846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2315.755291] Lustre: Skipped 1 previous similar message [ 2320.866606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2320.881677] Lustre: Skipped 1 previous similar message [ 2322.400915] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2322.481108] LustreError: 95670:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2322.487058] LustreError: 95670:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2322.523898] Lustre: server umount lustre-MDT0000 complete [ 2325.957288] LustreError: 95873:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2325.961106] LustreError: 95873:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2326.179600] Lustre: server umount lustre-MDT0001 complete [ 2329.811347] LustreError: 96073:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2329.815723] LustreError: 96073:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2329.872484] Lustre: server umount MGS complete [ 2341.168594] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:21:23 (1760127683) [ 2372.854286] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 2381.208223] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2391.235317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2398.009520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2404.802333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2411.277766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2419.133899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2419.252161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.117833] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2432.452741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2432.516083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2432.675179] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2432.710582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2432.769934] Lustre: lustre-MDT0001: new disk, initializing [ 2432.842884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2432.865044] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2432.868726] Lustre: 100592:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2432.878955] LustreError: 100592:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2436.897267] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2447.710527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2447.794662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2448.142445] Lustre: lustre-OST0000: new disk, initializing [ 2448.156816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2448.246692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.488229] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2456.162400] Lustre: 101764:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000007e4c5d59 x1845627524783744/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1760127816 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2460.380375] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2466.764422] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.773859] Lustre: MGS: Client 7f9005c3-8b23-4efc-a3e2-e9294fa2f5a1 (at 0@lo) reconnecting [ 2466.830326] Lustre: 99878:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2466.854978] LustreError: 99878:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2466.890685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2466.943539] Lustre: lustre-MDT0000: new disk, initializing [ 2467.083322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.112669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2467.215440] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2468.192841] Lustre: 102302:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000001dc15103 x1845627524790272/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1760127828 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2471.044989] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2476.526643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2476.533929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2477.285990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2477.361919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2501.091209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.099802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2501.103410] Lustre: Skipped 1 previous similar message [ 2501.113186] Lustre: Skipped 1 previous similar message [ 2511.333424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.340042] Lustre: Skipped 3 previous similar messages [ 2513.381022] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2513.425350] LustreError: 102823:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2513.428615] LustreError: 102823:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2513.693706] Lustre: server umount lustre-OST0000 complete [ 2521.575893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.582902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2531.296384] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2531.386389] LustreError: 103025:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2531.388529] LustreError: 103025:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2531.418456] Lustre: server umount lustre-MDT0000 complete [ 2531.820283] LustreError: 100563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.851316] LustreError: 100563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2549.216171] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2549.305455] LustreError: 103231:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2549.308724] LustreError: 103231:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2549.543726] Lustre: server umount lustre-MDT0001 complete [ 2552.268071] Lustre: server umount MGS complete [ 2561.148736] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:25:04 (1760127904) [ 2574.508081] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2576.833365] Key type lgssc unregistered [ 2577.107410] LNet: 104771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2577.122925] LNetError: 104771:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2577.134486] LNet: Removed LNI 192.168.202.119@tcp [ 2577.835636] Key type .llcrypt unregistered [ 2577.839164] Key type ._llcrypt unregistered [ 2592.355085] Key type ._llcrypt registered [ 2592.356378] Key type .llcrypt registered [ 2592.475319] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2593.252802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2593.270332] alg: No test for adler32 (adler32-zlib) [ 2594.262698] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 2594.448170] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2596.088199] Key type lgssc registered [ 2597.088739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2603.733281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2603.745806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.064804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.023737] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2615.271379] LustreError: 105972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.284318] LustreError: 105972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2615.655153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2615.972899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2619.232653] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2624.041956] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2627.690457] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2634.344756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2634.591572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.703131] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2645.844122] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2651.005525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2661.356189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.360183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2661.367621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2661.379076] Lustre: Skipped 1 previous similar message [ 2666.468795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2666.477793] Lustre: Skipped 2 previous similar messages [ 2671.072157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2671.118228] LustreError: 108216:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2671.249946] Lustre: server umount lustre-OST0000 complete [ 2676.714563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.718202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2676.741095] Lustre: Skipped 1 previous similar message [ 2676.762716] Lustre: Skipped 1 previous similar message [ 2681.130970] LustreError: 108418:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2681.145358] LustreError: 108418:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2681.306391] Lustre: server umount lustre-MDT0000 complete [ 2681.828233] LustreError: 107034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.838029] LustreError: 107034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2684.733342] LustreError: 105950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128028 with bad export cookie 15593852268255247561 [ 2684.736610] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2684.752457] LustreError: 105950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2684.907549] LustreError: 108621:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2684.913777] LustreError: 108621:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2685.234688] Lustre: server umount lustre-MDT0001 complete [ 2696.488738] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2699.511855] Key type lgssc unregistered [ 2699.817289] LNet: 109153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2699.832942] LNetError: 109153:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2699.862774] LNet: Removed LNI 192.168.202.119@tcp [ 2700.843185] Key type .llcrypt unregistered [ 2700.845137] Key type ._llcrypt unregistered [ 2717.224774] Key type ._llcrypt registered [ 2717.226625] Key type .llcrypt registered [ 2717.315847] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2718.197514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2718.310278] alg: No test for adler32 (adler32-zlib) [ 2719.409413] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 2719.634758] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2721.296209] Key type lgssc registered [ 2722.480105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2729.067553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2729.075751] LustreError: 110208:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2729.115178] LustreError: 110220:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2729.121753] LustreError: 110208:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4969 [ 2729.125259] LustreError: 110208:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2729.129383] LustreError: 110208:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2729.145370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.745799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.199085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2739.700521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2743.965100] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2748.730601] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing lsmod [ 2751.707205] LustreError: 111396:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2751.737373] LustreError: 111400:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2751.745498] LustreError: 111396:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 2751.754122] LustreError: 111396:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2751.763840] LustreError: 111396:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2751.775412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2752.200477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.043094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2755.055884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.089298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2757.602134] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.610651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2757.626731] Lustre: Skipped 1 previous similar message [ 2757.636196] Lustre: Skipped 1 previous similar message [ 2762.725823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.733518] Lustre: Skipped 3 previous similar messages [ 2764.784518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2764.794453] Lustre: Skipped 2 previous similar messages [ 2769.376826] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2769.461483] LustreError: 111587:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2769.565085] Lustre: server umount lustre-MDT0000 complete [ 2769.891493] LustreError: 110253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.912505] LustreError: 110253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2773.236899] LustreError: 110237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128117 with bad export cookie 8045932946950977559 [ 2773.247502] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2773.254290] LustreError: 110237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2775.009446] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.011611] LustreError: 111438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.026363] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2775.072862] LustreError: 110693:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000625dbb8a x1845628043810432/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2775.123401] LustreError: 111792:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2775.132489] LustreError: 111792:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2775.417114] Lustre: server umount lustre-MDT0001 complete [ 2779.506588] LustreError: 111994:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2779.510355] LustreError: 111994:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2779.676176] Lustre: server umount lustre-OST0000 complete [ 2789.516961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.068243] LustreError: 112470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.124355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.830062] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2795.489678] LustreError: 112470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.611922] LustreError: 112470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.261379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2805.601491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2810.448202] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2817.466343] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2822.050367] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2828.711283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2829.061102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.892900] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2835.260696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2841.033946] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2849.335252] LustreError: 114718:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2849.345973] LustreError: 114718:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2849.546715] Lustre: server umount lustre-OST0000 complete [ 2855.908544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.916974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2855.925970] Lustre: Skipped 1 previous similar message [ 2863.084773] LustreError: 114919:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2863.087658] LustreError: 114919:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2863.155200] Lustre: server umount lustre-MDT0000 complete [ 2866.081331] Lustre: 109621:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760128194/real 1760128194] req@0000000050ea2c70 x1845628043836672/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1760128210 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2866.137207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.157901] Lustre: Skipped 1 previous similar message [ 2866.167273] LustreError: 114158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2866.185865] LustreError: 114158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2866.457854] LustreError: 112449:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128210 with bad export cookie 8045932946950978259 [ 2866.463812] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2866.475534] LustreError: 112449:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2866.879729] Lustre: server umount lustre-MDT0001 complete [ 2876.622177] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2878.910185] Key type lgssc unregistered [ 2879.204958] LNet: 115656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2879.210762] LNetError: 115656:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2880.230678] LNet: Removed LNI 192.168.202.119@tcp [ 2881.237367] Key type .llcrypt unregistered [ 2881.240063] Key type ._llcrypt unregistered [ 2903.625905] Key type ._llcrypt registered [ 2903.627312] Key type .llcrypt registered [ 2903.715524] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2904.538941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2904.561580] alg: No test for adler32 (adler32-zlib) [ 2905.626748] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 2905.800103] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2907.464208] Key type lgssc registered [ 2908.679069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.522776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2915.552524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.073257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2920.526165] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2927.591226] LustreError: 117052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.597546] LustreError: 117052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2927.709893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2928.145119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2931.359426] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2935.977622] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2939.571849] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2946.101047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2946.508917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.196699] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2957.519608] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2962.968780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2966.620262] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2969.202580] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2973.158717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2973.173818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.176682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2977.977201] LustreError: 119399:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2978.145911] Lustre: server umount lustre-OST0000 complete [ 2983.405542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.412712] Lustre: Skipped 2 previous similar messages [ 2983.416140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2983.418936] Lustre: Skipped 1 previous similar message [ 2987.301144] LustreError: 119604:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2987.308035] LustreError: 119604:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2987.369080] Lustre: server umount lustre-MDT0000 complete [ 2988.514352] LustreError: 117052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.529144] LustreError: 117052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2990.788892] LustreError: 117032:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128334 with bad export cookie 11728002058124012216 [ 2990.791663] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2990.796452] LustreError: 117032:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2990.953076] LustreError: 119805:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2990.961400] LustreError: 119805:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2991.165255] Lustre: server umount lustre-MDT0001 complete [ 2999.592424] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3001.861579] Key type lgssc unregistered [ 3002.087913] LNet: 120338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3002.101672] LNetError: 120338:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3003.114882] LNet: Removed LNI 192.168.202.119@tcp [ 3003.940349] Key type .llcrypt unregistered [ 3003.945274] Key type ._llcrypt unregistered [ 3017.758402] Key type ._llcrypt registered [ 3017.760165] Key type .llcrypt registered [ 3017.850983] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3018.484815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3018.508870] alg: No test for adler32 (adler32-zlib) [ 3019.544704] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3019.722780] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3021.392236] Key type lgssc registered [ 3022.463330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3028.984174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3029.009996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.390156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3033.547683] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3040.163255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3040.439995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3043.635496] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3048.383621] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3051.739574] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3056.564320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3056.773444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.445935] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3062.254314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3064.052845] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3070.433886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3070.435449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.445670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3070.449220] Lustre: Skipped 1 previous similar message [ 3071.543106] LustreError: 123768:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3071.657575] Lustre: server umount lustre-OST0000 complete [ 3075.556295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.563387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3075.566931] Lustre: Skipped 2 previous similar messages [ 3080.291745] LustreError: 123969:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3080.299848] LustreError: 123969:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3080.400882] Lustre: server umount lustre-MDT0000 complete [ 3080.674640] LustreError: 122728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.684561] LustreError: 122728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3082.717206] LustreError: 123970:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128426 with bad export cookie 4655456188950860057 [ 3082.723704] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3082.727709] LustreError: 123970:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3082.822746] LustreError: 124171:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3082.826080] LustreError: 124171:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3082.979806] Lustre: server umount lustre-MDT0001 complete [ 3090.414408] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3092.462953] Key type lgssc unregistered [ 3092.675764] LNet: 124704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3092.689209] LNetError: 124704:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3092.699948] LNet: Removed LNI 192.168.202.119@tcp [ 3093.318309] Key type .llcrypt unregistered [ 3093.319898] Key type ._llcrypt unregistered [ 3110.724837] Key type ._llcrypt registered [ 3110.726549] Key type .llcrypt registered [ 3110.783682] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3111.235154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3111.249904] alg: No test for adler32 (adler32-zlib) [ 3112.183652] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3112.323701] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3113.944591] Key type lgssc registered [ 3114.730465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3121.048826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3121.066506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.386900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3125.188123] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3131.992203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3132.321581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3134.995806] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3138.920493] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3141.451169] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3146.272780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3146.505340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3148.595878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3150.755615] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3154.666719] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3163.772574] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3166.575588] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3166.579436] Lustre: Skipped 2 previous similar messages [ 3170.123987] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3170.130799] Lustre: Skipped 2 previous similar messages [ 3173.413171] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3173.432367] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3180.365267] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.19@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3180.373369] Lustre: Skipped 2 previous similar messages [ 3181.095650] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.19@tcp 192.168.0.[2,10]@tcp [ 3189.210935] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.19/24@tcp in log lustre-MDT0000 [ 3189.220823] Lustre: Skipped 2 previous similar messages [ 3197.982760] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.19/24@tcp [ 3197.993112] Lustre: Skipped 1 previous similar message [ 3204.071940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.072565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3204.075046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.076655] Lustre: Skipped 1 previous similar message [ 3208.692962] LustreError: 129776:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3208.808035] Lustre: server umount lustre-OST0000 complete [ 3214.311697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.311857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3214.327648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3214.330071] Lustre: Skipped 2 previous similar messages [ 3217.761398] LustreError: 129978:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3217.764342] LustreError: 129978:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3217.872517] Lustre: server umount lustre-MDT0000 complete [ 3219.427107] LustreError: 126081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.435122] LustreError: 126081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3220.578717] LustreError: 126063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128564 with bad export cookie 14594877726201022991 [ 3220.582266] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3220.592878] LustreError: 126063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3220.703184] LustreError: 130180:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3220.712378] LustreError: 130180:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3220.900656] Lustre: server umount lustre-MDT0001 complete [ 3228.470964] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3230.572637] Key type lgssc unregistered [ 3230.808827] LNet: 130712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3230.815229] LNetError: 130712:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3231.851938] LNet: Removed LNI 192.168.202.119@tcp [ 3232.376510] Key type .llcrypt unregistered [ 3232.379120] Key type ._llcrypt unregistered [ 3250.499455] Key type ._llcrypt registered [ 3250.501138] Key type .llcrypt registered [ 3250.571419] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3251.040525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3251.116244] alg: No test for adler32 (adler32-zlib) [ 3252.052473] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3252.202851] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3253.816952] Key type lgssc registered [ 3254.461689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3259.624613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3259.641088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.938640] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3260.954919] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.19/24@tcp [ 3260.960628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.595936] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3269.405783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3269.648783] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3269.672833] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.19/24@tcp [ 3269.692624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3272.646454] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3276.119724] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3278.657715] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3284.387650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3284.720157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.796880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3289.345277] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3293.295949] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3305.443069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3305.448114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.452469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.453615] Lustre: Skipped 1 previous similar message [ 3309.947328] LustreError: 134530:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3310.113543] Lustre: server umount lustre-OST0000 complete [ 3315.682208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3315.682789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.684902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.684909] Lustre: Skipped 1 previous similar message [ 3315.701204] Lustre: Skipped 1 previous similar message [ 3319.454479] LustreError: 134731:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3319.457967] LustreError: 134731:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3319.557343] Lustre: server umount lustre-MDT0000 complete [ 3320.809035] LustreError: 132226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.824284] LustreError: 132226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3322.274611] LustreError: 132211:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128666 with bad export cookie 14317671661808735679 [ 3322.280254] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.285664] LustreError: 132211:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3322.382639] LustreError: 134934:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3322.386481] LustreError: 134934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3322.553929] Lustre: server umount lustre-MDT0001 complete [ 3330.569428] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3332.636750] Key type lgssc unregistered [ 3332.869687] LNet: 135467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3332.875155] LNetError: 135467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3332.889050] LNet: Removed LNI 192.168.202.119@tcp [ 3333.481082] Key type .llcrypt unregistered [ 3333.482537] Key type ._llcrypt unregistered [ 3351.542989] Key type ._llcrypt registered [ 3351.544924] Key type .llcrypt registered [ 3351.630905] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3363.200637] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3364.148550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3364.207709] alg: No test for adler32 (adler32-zlib) [ 3365.207054] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3365.385285] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3367.032383] Key type lgssc registered [ 3367.816541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3374.488471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3379.853314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3384.753679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3389.948720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3395.650121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3395.705378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3395.717491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.892476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3396.920941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3397.024690] Lustre: lustre-MDT0000: new disk, initializing [ 3397.130424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.158022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3400.162328] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3407.544243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3407.611846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3407.705767] Lustre: 138821:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3407.772588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3407.775944] Lustre: Skipped 1 previous similar message [ 3407.836350] Lustre: lustre-MDT0001: new disk, initializing [ 3407.917275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3407.950619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3407.955510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3410.976707] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3416.051727] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3418.358933] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3423.454268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3423.498762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3423.701511] Lustre: lustre-OST0000: new disk, initializing [ 3423.709878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3423.788236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.134651] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3428.918604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3428.927576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3429.022688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3433.244375] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3436.014848] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3436.161637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3438.705066] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3438.834836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3444.195538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3444.201220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3444.209749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3444.708795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3444.730305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3446.078137] LustreError: 141758:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3446.199583] Lustre: server umount lustre-OST0000 complete [ 3449.826850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.839205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.842435] Lustre: Skipped 1 previous similar message [ 3454.947242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.952326] Lustre: Skipped 1 previous similar message [ 3455.114392] LustreError: 141960:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3455.116486] LustreError: 141960:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3455.155342] Lustre: server umount lustre-MDT0000 complete [ 3457.625651] LustreError: 138812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128801 with bad export cookie 17996991549101551915 [ 3457.628681] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3457.646343] LustreError: 138812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3457.765897] LustreError: 142162:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3457.769849] LustreError: 142162:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3457.947478] Lustre: server umount lustre-MDT0001 complete [ 3463.797421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.166452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3466.679578] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3469.284046] LustreError: 142637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.290324] LustreError: 142637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3472.512087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3472.774544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3475.038279] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3478.416068] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3480.625511] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3484.915833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3485.143707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.861625] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3492.889083] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3495.825394] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3495.967751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3498.932933] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3499.154167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3506.947123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3507.012289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3507.130576] Lustre: lustre-OST0001: new disk, initializing [ 3507.133387] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3507.196425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3511.769700] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3514.952887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3514.962284] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3515.058639] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3518.302486] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3518.594391] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3522.120539] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3522.315702] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3525.203840] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88d642c7f000.ost_server_uuid 50 [ 3526.559945] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88d642c7f000.ost_server_uuid in FULL state after 0 sec [ 3533.684121] LustreError: 146882:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3533.687326] LustreError: 146882:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3533.739837] Lustre: server umount lustre-OST0001 complete [ 3535.843348] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.852033] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.856544] LustreError: 144235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.861640] Lustre: Skipped 2 previous similar messages [ 3535.883804] LustreError: 144235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3538.407138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3538.410424] LustreError: Skipped 1 previous similar message [ 3538.418397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3540.448983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3540.454428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.461763] Lustre: Skipped 1 previous similar message [ 3540.968867] LustreError: 144234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3540.982152] LustreError: 144234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3542.650158] LustreError: 147083:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3542.652311] LustreError: 147083:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3542.745882] Lustre: server umount lustre-OST0000 complete [ 3551.204077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3551.208224] Lustre: Skipped 5 previous similar messages [ 3551.717622] LustreError: 147284:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3551.726814] LustreError: 147284:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3551.827232] Lustre: server umount lustre-MDT0000 complete [ 3554.548822] LustreError: 142617:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128898 with bad export cookie 17996991549101552804 [ 3554.559119] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3554.844688] Lustre: server umount lustre-MDT0001 complete [ 3570.044547] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3572.643827] Key type lgssc unregistered [ 3572.915767] LNet: 148476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3572.925285] LNetError: 148476:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3572.945796] LNet: Removed LNI 192.168.202.119@tcp [ 3573.697232] Key type .llcrypt unregistered [ 3573.699845] Key type ._llcrypt unregistered [ 3599.337342] Key type ._llcrypt registered [ 3599.338979] Key type .llcrypt registered [ 3599.425482] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3611.214993] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3612.181550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3612.204075] alg: No test for adler32 (adler32-zlib) [ 3613.252343] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3613.513886] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3615.168238] Key type lgssc registered [ 3616.415344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3624.830987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3630.531386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3635.901071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3641.686989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3647.132272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3647.178884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3647.193688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.376322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3648.397753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3648.453454] Lustre: lustre-MDT0000: new disk, initializing [ 3648.535937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3648.553699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3651.586724] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3660.118708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3660.169395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3660.241219] Lustre: 152403:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3660.282907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3660.285807] Lustre: Skipped 1 previous similar message [ 3660.339557] Lustre: lustre-MDT0001: new disk, initializing [ 3660.443855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3660.497730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3660.508250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3663.496855] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3668.625136] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3671.086954] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3676.251912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3676.296736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3676.520703] Lustre: lustre-OST0000: new disk, initializing [ 3676.531431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3676.616149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.830694] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3683.907469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3683.922667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3684.009739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3686.484154] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3689.653333] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3689.827230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3693.083746] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3693.266197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3699.684276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.689068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3699.711741] Lustre: Skipped 1 previous similar message [ 3699.727744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3700.982583] LustreError: 155342:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3701.146034] Lustre: server umount lustre-OST0000 complete [ 3704.802469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.806102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3704.819130] Lustre: Skipped 1 previous similar message [ 3704.825866] Lustre: Skipped 2 previous similar messages [ 3709.926963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3709.934396] Lustre: Skipped 1 previous similar message [ 3710.444415] LustreError: 155543:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3710.448108] LustreError: 155543:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3710.520954] Lustre: server umount lustre-MDT0000 complete [ 3713.328441] LustreError: 152396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760129057 with bad export cookie 11464963604435908897 [ 3713.333942] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.360586] LustreError: 152396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3713.471141] LustreError: 155744:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3713.473475] LustreError: 155744:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3713.637509] Lustre: server umount lustre-MDT0001 complete [ 3719.077971] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 16:44:22 (1760129062) [ 3731.391022] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3736.547179] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3742.251153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3747.072255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3752.212822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3757.672661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3763.111260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3763.165860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.345733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3763.372204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3763.447983] Lustre: lustre-MDT0000: new disk, initializing [ 3763.525515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3763.540621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3766.680719] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3775.729487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3775.792582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3775.869245] Lustre: 158736:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3775.942616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3775.946062] Lustre: Skipped 1 previous similar message [ 3776.013455] Lustre: lustre-MDT0001: new disk, initializing [ 3776.097035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3776.131813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3776.149437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3779.372550] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3785.621819] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3788.204650] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3793.069641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3793.127487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3793.321315] Lustre: lustre-OST0000: new disk, initializing [ 3793.329202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3793.388839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3797.914862] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3801.123807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3801.129634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3801.223413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3803.248273] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3807.126301] Lustre: Failing over lustre-OST0000 [ 3809.203266] LustreError: 161175:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3809.209797] LustreError: 161175:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3809.296680] Lustre: server umount lustre-OST0000 complete [ 3824.389161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3824.580134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.586208] Lustre: Skipped 1 previous similar message [ 3824.598763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.607702] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3825.632153] Lustre: 150614:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760129153/real 1760129153] req@000000002156a703 x1845628981280896/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1760129169 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3826.104949] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3827.664246] Lustre: 150617:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760129155/real 1760129155] req@0000000077ee9eef x1845628981281024/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1760129171 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3827.789978] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3829.756368] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 3829.759452] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3830.395223] Lustre: Failing over lustre-MDT0000 [ 3830.481257] LustreError: 162151:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3830.483815] LustreError: 162151:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3832.291841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3832.295580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.480139] Lustre: 150614:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760129160/real 1760129160] req@0000000034efa139 x1845628981282816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760129176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3832.502614] Lustre: 150614:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3832.577189] Lustre: server umount lustre-MDT0000 complete [ 3836.327505] LustreError: 158741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.338859] LustreError: 158741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3838.944521] Lustre: 150614:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760129166/real 1760129166] req@00000000aba1b91d x1845628981283840/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760129182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3839.975254] LustreError: 159897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3839.993821] LustreError: 159897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3841.446850] LustreError: 160987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.091400] LustreError: 159897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.100848] LustreError: 159897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3847.973479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3848.048322] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3848.056743] Lustre: 158738:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.119@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3848.241854] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3850.961751] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3851.690254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3853.288774] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3853.294268] Lustre: Skipped 1 previous similar message [ 3853.309232] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3853.339921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3858.405339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.406480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3858.412194] Lustre: Skipped 3 previous similar messages [ 3860.276667] LustreError: 163036:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3860.279527] LustreError: 163036:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3860.386814] Lustre: server umount lustre-OST0000 complete [ 3863.524281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3863.525285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3863.527579] Lustre: Skipped 2 previous similar messages [ 3873.761699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3873.764781] Lustre: Skipped 3 previous similar messages [ 3877.345206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3877.428375] LustreError: 163237:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3877.440724] LustreError: 163237:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3877.540481] Lustre: server umount lustre-MDT0000 complete [ 3878.882386] LustreError: 159897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3878.893324] LustreError: 159897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3880.121047] LustreError: 160480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760129224 with bad export cookie 11464963604435910990 [ 3880.123186] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.126631] LustreError: 160480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3880.400651] Lustre: server umount lustre-MDT0001 complete [ 3886.820682] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3888.709334] Key type lgssc unregistered [ 3888.930454] LNet: 163973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3888.939351] LNetError: 163973:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3888.955531] LNet: Removed LNI 192.168.202.119@tcp [ 3889.592620] Key type .llcrypt unregistered [ 3889.594219] Key type ._llcrypt unregistered [ 3917.992605] Key type ._llcrypt registered [ 3917.993946] Key type .llcrypt registered [ 3918.065931] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3918.849757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3918.920853] alg: No test for adler32 (adler32-zlib) [ 3919.902687] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3920.074354] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3921.680182] Key type lgssc registered [ 3922.547811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3927.543487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3927.560391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.847498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.102360] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3936.142410] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3938.280848] LustreError: 165473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.293965] LustreError: 165473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3939.252107] LustreError: 165502:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3939.267753] LustreError: 165502:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3939.279404] LustreError: 165502:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 3939.381917] LustreError: 166025:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3939.523594] Lustre: server umount lustre-MDT0000 complete [ 3949.551178] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3954.852932] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3961.449590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3966.351892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3970.425450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3974.563781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3979.223575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3979.271215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.401385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3979.431360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3979.491171] Lustre: lustre-MDT0000: new disk, initializing [ 3979.560319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3979.572991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3982.136534] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3989.329309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3989.371203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3989.427261] Lustre: 168666:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3989.466474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3989.469174] Lustre: Skipped 1 previous similar message [ 3989.515796] Lustre: lustre-MDT0001: new disk, initializing [ 3989.579239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3989.606513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3989.616257] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3991.942619] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3997.006386] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3999.084705] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4002.839758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4002.879362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4003.037181] Lustre: lustre-OST0000: new disk, initializing [ 4003.041026] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4003.092935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4003.367162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4003.371414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4003.438882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4006.566845] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4011.943690] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4014.570909] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4015.561884] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:49:19 (1760129359) [ 4197.856964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.858445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4197.862590] Lustre: Skipped 1 previous similar message [ 4197.873470] Lustre: Skipped 2 previous similar messages [ 4202.976830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4202.986881] Lustre: Skipped 2 previous similar messages [ 4208.098219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.102101] Lustre: Skipped 2 previous similar messages [ 4209.122508] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4209.174710] LustreError: 172184:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4209.176859] LustreError: 172184:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4209.256747] Lustre: server umount lustre-MDT0000 complete [ 4211.388899] LustreError: 170414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760129555 with bad export cookie 3120902893805439875 [ 4211.389650] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.394265] LustreError: 170414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4213.216623] LustreError: 171023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4213.216762] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.233905] Lustre: Skipped 1 previous similar message [ 4213.237272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4216.225250] LustreError: 171023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4217.550797] LustreError: 172389:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4217.553688] LustreError: 172389:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4217.721550] Lustre: server umount lustre-MDT0001 complete [ 4234.208144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4234.238887] LustreError: 172590:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4234.243230] LustreError: 172590:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4234.399594] Lustre: server umount lustre-OST0000 complete [ 4240.577790] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4246.314147] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4252.639976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4256.830205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4260.873342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4265.012204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4270.065670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4270.119135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4270.306825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4270.344799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4270.410897] Lustre: lustre-MDT0000: new disk, initializing [ 4270.505437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4270.520129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4273.309261] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4280.920289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4280.961654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4281.017716] Lustre: 174929:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4281.053151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4281.060049] Lustre: Skipped 1 previous similar message [ 4281.113461] Lustre: lustre-MDT0001: new disk, initializing [ 4281.168483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4281.195181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4281.200615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4283.440615] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4288.261104] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4290.497459] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4294.307056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4294.342554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4294.506672] Lustre: lustre-OST0000: new disk, initializing [ 4294.511722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4298.542530] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4300.310122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4300.314585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4300.378549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4303.359093] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4305.716673] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4305.858742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4308.025304] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4308.151428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4310.497821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4310.505110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.511343] LustreError: Skipped 1 previous similar message [ 4310.518737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4310.518745] Lustre: Skipped 1 previous similar message [ 4315.343279] LustreError: 177868:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4315.353254] LustreError: 177868:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4315.469477] Lustre: server umount lustre-OST0000 complete [ 4320.737350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4320.739525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4320.745415] Lustre: Skipped 1 previous similar message [ 4320.746126] Lustre: Skipped 2 previous similar messages [ 4323.661934] LustreError: 178069:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4323.665792] LustreError: 178069:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4323.729600] Lustre: server umount lustre-MDT0000 complete [ 4325.857420] LustreError: 174941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.863882] LustreError: 174941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4325.916549] LustreError: 174922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760129669 with bad export cookie 3120902893805598705 [ 4325.920685] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.922699] LustreError: 174922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4326.147967] Lustre: server umount lustre-MDT0001 complete [ 4331.076124] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:54:34 (1760129674) [ 4340.282674] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4345.216914] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4350.596766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4354.647938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4357.934790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4360.959790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.280956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.334097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.484170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4364.500893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4364.549248] Lustre: lustre-MDT0000: new disk, initializing [ 4364.612952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4364.616236] Lustre: Skipped 1 previous similar message [ 4364.628694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4366.789953] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4373.253461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4373.294226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4373.357921] Lustre: 181257:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4373.555937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4375.718676] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4380.706068] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4382.641289] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4386.160709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4386.194140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4389.371590] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4392.496342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4392.570801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4393.700251] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4397.539967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.541100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4397.542569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4397.542574] Lustre: Skipped 1 previous similar message [ 4397.555236] Lustre: Skipped 1 previous similar message [ 4399.029110] LustreError: 183803:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4399.031756] LustreError: 183803:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4399.166572] Lustre: server umount lustre-OST0000 complete [ 4407.500394] Lustre: server umount lustre-MDT0000 complete [ 4407.778605] LustreError: 181267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.791105] LustreError: 181267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4409.461112] LustreError: 183716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760129753 with bad export cookie 3120902893805599587 [ 4409.464562] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4409.469566] LustreError: 183716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4409.668349] Lustre: server umount lustre-MDT0001 complete [ 4414.367065] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:55:58 (1760129758) [ 4422.536987] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4427.276636] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4432.147301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4435.482520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4438.870954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4442.178526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4446.021428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4446.061253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.178367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4446.198423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4446.200916] Lustre: Skipped 3 previous similar messages [ 4446.247599] Lustre: lustre-MDT0000: new disk, initializing [ 4446.249542] Lustre: Skipped 2 previous similar messages [ 4446.303121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.305790] Lustre: Skipped 2 previous similar messages [ 4446.322145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4446.326189] Lustre: Skipped 2 previous similar messages [ 4448.299707] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4454.289603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4454.326559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4454.384878] Lustre: 187189:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4454.574959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4456.755314] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4461.120205] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4463.020778] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4466.534459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4466.591470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4469.780915] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4473.935062] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4475.479070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4485.603080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4485.603468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.606219] LustreError: Skipped 1 previous similar message [ 4485.612055] Lustre: Skipped 3 previous similar messages [ 4485.614906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4485.617249] Lustre: Skipped 4 previous similar messages [ 4488.444171] LustreError: 189734:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4488.448069] LustreError: 189734:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4488.533691] Lustre: server umount lustre-OST0000 complete [ 4495.841272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4497.075453] Lustre: server umount lustre-MDT0000 complete [ 4499.040209] LustreError: 187975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760129843 with bad export cookie 3120902893805600672 [ 4499.046346] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4499.048818] LustreError: 187975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4499.232181] Lustre: server umount lustre-MDT0001 complete [ 4504.864294] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 4506.547732] Key type lgssc unregistered [ 4506.747268] LNet: 190668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4506.751116] LNetError: 190668:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4506.757929] LNet: Removed LNI 192.168.202.119@tcp [ 4507.287788] Key type .llcrypt unregistered [ 4507.289493] Key type ._llcrypt unregistered [ 4521.179846] Key type ._llcrypt registered [ 4521.181568] Key type .llcrypt registered [ 4521.256949] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4521.746145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4521.810140] alg: No test for adler32 (adler32-zlib) [ 4522.769664] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 4522.904255] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 4524.520229] Key type lgssc registered [ 4525.131822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4529.402529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4529.429331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.724509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.132577] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4537.683298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4537.896948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4539.981800] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4542.775930] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4544.647921] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4548.267946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4548.498213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4551.592283] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4554.438527] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4555.690066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4578.277306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4578.277906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.279432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4578.290697] Lustre: Skipped 1 previous similar message [ 4583.158196] LustreError: 194313:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4583.247981] Lustre: server umount lustre-OST0000 complete [ 4588.514194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.517155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4588.519581] Lustre: Skipped 1 previous similar message [ 4588.521980] Lustre: Skipped 1 previous similar message [ 4591.501244] LustreError: 194515:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4591.504553] LustreError: 194515:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4591.557608] Lustre: server umount lustre-MDT0000 complete [ 4593.554888] LustreError: 192025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760129937 with bad export cookie 14074433780728297571 [ 4593.559771] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4593.566272] LustreError: 192025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4593.672413] LustreError: 194718:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4593.677310] LustreError: 194718:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4593.802954] Lustre: server umount lustre-MDT0001 complete [ 4598.785166] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 4600.104843] Key type lgssc unregistered [ 4600.254951] LNet: 195250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4600.258585] LNetError: 195250:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4600.270649] LNet: Removed LNI 192.168.202.119@tcp [ 4600.643558] Key type .llcrypt unregistered [ 4600.645081] Key type ._llcrypt unregistered [ 4612.522786] Key type ._llcrypt registered [ 4612.524360] Key type .llcrypt registered [ 4612.598371] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4613.054251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4613.100103] alg: No test for adler32 (adler32-zlib) [ 4614.064510] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 4614.204403] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 4615.842836] Key type lgssc registered [ 4616.432566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4620.327864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4620.337686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.589882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4623.537953] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4627.779319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4627.984319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4630.115782] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4632.883230] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4634.595296] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4638.226574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4638.400635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.436065] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4644.765527] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4646.637600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4662.242533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4662.245672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4662.253081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4663.268250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4663.268692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4663.272292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.281950] Lustre: Skipped 2 previous similar messages [ 4665.140147] LustreError: 198893:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4665.242734] Lustre: server umount lustre-OST0000 complete [ 4668.110867] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88d644b1f800.ost_server_uuid 50 [ 4669.021834] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4680.157381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.162618] Lustre: Skipped 1 previous similar message [ 4680.166721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.170046] Lustre: Skipped 1 previous similar message [ 4683.234912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4683.238236] Lustre: Skipped 1 previous similar message [ 4686.300649] LustreError: 199255:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4686.304123] LustreError: 199255:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4686.357802] Lustre: server umount lustre-MDT0000 complete [ 4688.353554] LustreError: 196629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.361288] LustreError: 196629:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4688.375401] LustreError: 196610:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130032 with bad export cookie 161733465415761293 [ 4688.378581] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4688.380765] LustreError: 196610:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4688.453054] LustreError: 199457:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4688.455723] LustreError: 199457:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4688.560203] Lustre: server umount lustre-MDT0001 complete [ 4692.644382] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:00:36 (1760130036) [ 4695.941058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.166547] LustreError: 200127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4696.217918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4698.067076] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4701.665898] LustreError: 200127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.542904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4702.762676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4704.899571] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4707.653899] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4709.369286] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4712.511220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4712.684624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.286820] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4718.318866] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4721.900573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4722.069947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4722.109628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4722.207376] Lustre: lustre-OST0001: new disk, initializing [ 4722.210482] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4725.133831] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4727.847120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4727.853960] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4727.895907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4729.288456] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4732.387089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4732.391268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4732.400264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4736.828169] LustreError: 203339:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4736.833129] LustreError: 203339:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4736.866912] Lustre: server umount lustre-OST0000 complete [ 4738.021122] LustreError: 201722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.033101] LustreError: 201722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4739.841969] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4739.957934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4742.002517] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4742.134207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4743.138902] LustreError: 201724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4743.145816] LustreError: 201724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4753.378561] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4753.379949] LustreError: 201722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.380093] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.380099] Lustre: Skipped 1 previous similar message [ 4753.382646] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4753.382652] Lustre: Skipped 2 previous similar messages [ 4753.385514] LustreError: Skipped 1 previous similar message [ 4753.390764] LustreError: 201722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4758.962125] LustreError: 204042:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4758.967478] LustreError: 204042:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4759.047393] Lustre: server umount lustre-OST0001 complete [ 4763.617536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4763.622542] Lustre: Skipped 1 previous similar message [ 4773.857978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4773.860922] Lustre: Skipped 8 previous similar messages [ 4775.392138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4775.435556] LustreError: 204243:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4775.438305] LustreError: 204243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4775.477435] Lustre: server umount lustre-MDT0000 complete [ 4777.141475] LustreError: 200108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130121 with bad export cookie 161733465415762350 [ 4777.143587] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4777.146596] LustreError: 200108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4777.316927] Lustre: server umount lustre-MDT0001 complete [ 4780.128228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4782.046129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4783.891169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4785.849102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4789.097692] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:02:12 (1760130132) [ 4792.465741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.624415] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4792.644327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4792.738453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.741692] Lustre: Skipped 1 previous similar message [ 4794.545831] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4798.462964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4798.525116] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4798.548814] Lustre: 205937:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4800.383612] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4802.794570] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4804.641633] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4807.637118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4807.737853] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4810.396571] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4810.979078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4812.852735] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4816.064590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4816.121996] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4818.839585] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4818.996136] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4821.390560] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4824.039025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4824.044248] Lustre: Skipped 1 previous similar message [ 4824.047571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4824.050242] Lustre: Skipped 1 previous similar message [ 4828.655511] LustreError: 209132:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4828.659908] LustreError: 209132:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4828.686654] Lustre: server umount lustre-OST0000 complete [ 4829.156693] LustreError: 207563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4829.161980] LustreError: 207563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4844.516338] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4844.520853] LustreError: Skipped 1 previous similar message [ 4846.342733] Lustre: server umount lustre-OST0001 complete [ 4854.320120] Lustre: server umount lustre-MDT0000 complete [ 4856.062576] LustreError: 205929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130200 with bad export cookie 161733465415763736 [ 4856.065399] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4856.069912] LustreError: 205929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4856.258440] Lustre: server umount lustre-MDT0001 complete [ 4859.116680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4861.086629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4863.142480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4865.208353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4868.510410] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:03:32 (1760130212) [ 4875.844655] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4879.728779] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4884.258908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4887.604355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4890.594195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4893.503503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4897.070663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4897.102511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4897.231830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4897.249373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4897.294404] Lustre: lustre-MDT0000: new disk, initializing [ 4897.344509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4897.347016] Lustre: Skipped 3 previous similar messages [ 4897.355291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4899.220364] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4905.021928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4905.056204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4905.102525] Lustre: 213557:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4905.131829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4905.135311] Lustre: Skipped 1 previous similar message [ 4905.177215] Lustre: lustre-MDT0001: new disk, initializing [ 4905.243305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4905.248085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4906.945474] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4910.795138] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4912.354691] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4915.392389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4915.427543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4915.553617] Lustre: lustre-OST0000: new disk, initializing [ 4915.557465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4918.300937] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4922.244794] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4922.384530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4922.390189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4922.444038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4924.140188] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4924.247837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4926.065345] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4926.166244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4927.458747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4927.458877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.462215] LustreError: Skipped 1 previous similar message [ 4927.463121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4927.467588] Lustre: Skipped 6 previous similar messages [ 4927.469601] Lustre: Skipped 5 previous similar messages [ 4929.077255] LustreError: 216496:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4929.082156] LustreError: 216496:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4929.173237] Lustre: server umount lustre-OST0000 complete [ 4945.376531] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4945.458226] Lustre: server umount lustre-MDT0000 complete [ 4947.016799] LustreError: 215305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130290 with bad export cookie 161733465415765500 [ 4947.021354] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.021761] LustreError: 215305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4947.193262] Lustre: server umount lustre-MDT0001 complete [ 4950.985396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.209486] LustreError: 217376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4951.218509] LustreError: 217376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4952.965517] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4957.017616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4958.887300] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4961.419263] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4963.069135] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4966.038302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4966.215644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.218510] Lustre: Skipped 4 previous similar messages [ 4968.800703] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4971.147573] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4972.780634] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4977.006198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4978.706338] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4978.815659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4981.735566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4981.739948] LustreError: Skipped 1 previous similar message [ 4985.786909] Lustre: server umount lustre-OST0000 complete [ 4988.144788] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4989.847238] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5014.546763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5017.453280] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5019.958877] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5020.773976] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5021.861796] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.868348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.868394] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5027.297144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5027.298208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5027.302619] LustreError: Skipped 1 previous similar message [ 5027.311133] Lustre: Skipped 5 previous similar messages [ 5028.333718] LustreError: 221780:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5028.336246] LustreError: 221780:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5028.408095] Lustre: server umount lustre-OST0000 complete [ 5036.336214] Lustre: server umount lustre-MDT0000 complete [ 5037.538217] LustreError: 217376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5037.550685] LustreError: 217376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5037.984850] LustreError: 217357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130381 with bad export cookie 161733465415766375 [ 5037.987366] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5037.989952] LustreError: 217357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5038.153211] Lustre: server umount lustre-MDT0001 complete [ 5041.807587] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:06:25 (1760130385) [ 5044.737955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.003902] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5050.934603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5052.736991] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5055.375402] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5056.993559] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5060.097830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5062.915699] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5065.348277] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5067.174390] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5070.386954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5072.153848] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5072.244980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5075.182609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5075.220534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5075.300028] Lustre: lustre-OST0001: new disk, initializing [ 5075.306346] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5077.792684] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5081.271358] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5082.152441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5082.156550] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5082.197166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5083.150511] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5083.259231] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5085.038796] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5085.139741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5087.211783] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5087.214982] Lustre: Skipped 15 previous similar messages [ 5088.020305] Lustre: server umount lustre-OST0001 complete [ 5090.702263] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5090.810951] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5092.506732] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5092.614121] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5116.663315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5116.749760] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5116.753228] Lustre: Skipped 5 previous similar messages [ 5119.054274] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5121.123849] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5121.764221] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5122.023713] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5122.029324] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5122.029613] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5122.037824] Lustre: Skipped 2 previous similar messages [ 5124.608882] Lustre: server umount lustre-OST0001 complete [ 5127.142793] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5127.146174] LustreError: Skipped 3 previous similar messages [ 5143.304543] Lustre: server umount lustre-OST0000 complete [ 5159.392123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5159.451648] LustreError: 229155:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5159.454727] LustreError: 229155:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 5159.514876] Lustre: server umount lustre-MDT0000 complete [ 5161.300201] LustreError: 222837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130505 with bad export cookie 161733465415767320 [ 5161.304343] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.304990] LustreError: 222837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5167.543654] Lustre: server umount lustre-MDT0001 complete [ 5170.068260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5172.029313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5173.745018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5175.489190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5178.433622] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:08:42 (1760130522) [ 5181.226959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.352182] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5181.366696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5183.040570] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5186.921967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5186.979062] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5186.995836] Lustre: 230849:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5188.883760] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5191.408636] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5192.956910] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5195.862922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5195.955706] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5198.612196] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5201.037727] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5210.981332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5211.046208] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5213.747588] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5216.249147] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5220.079457] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 5221.830386] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5221.926000] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5223.624249] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5223.718891] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5225.189439] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88d64561f800.ost_server_uuid 50 [ 5225.866662] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88d64561f800.ost_server_uuid in FULL state after 0 sec [ 5226.338752] Lustre: Permanently deactivating lustre-OST0001 [ 5226.346161] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5227.084878] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5227.556644] Lustre: Permanently reactivating lustre-OST0001 [ 5233.150823] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.156822] Lustre: Skipped 10 previous similar messages [ 5234.966370] Lustre: server umount lustre-OST0001 complete [ 5238.246306] LustreError: 232473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.254735] LustreError: 232473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 5243.018959] Lustre: server umount lustre-OST0000 complete [ 5253.600481] LustreError: 234851:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 5259.232134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5259.305995] Lustre: server umount lustre-MDT0000 complete [ 5260.949484] LustreError: 230840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130604 with bad export cookie 161733465415768580 [ 5260.953141] LustreError: 230840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5260.954387] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5261.123888] Lustre: server umount lustre-MDT0001 complete [ 5263.471247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5265.164743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5266.841130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5268.512899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5271.310288] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:10:15 (1760130615) [ 5271.750483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5274.587165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5274.724290] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5274.738747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5274.741840] Lustre: Skipped 5 previous similar messages [ 5276.475972] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5280.151569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5280.222962] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5280.242091] Lustre: 236999:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5282.072728] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5284.415658] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5286.000785] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5288.997589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5289.037279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5289.155715] Lustre: Permanently deactivating lustre-OST0000 [ 5289.162974] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5291.788894] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5294.048307] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5296.670667] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5296.676753] Lustre: Skipped 1 previous similar message [ 5297.077590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5299.655953] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5302.154701] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5305.396025] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5308.344538] Lustre: Permanently reactivating lustre-OST0000 [ 5308.352136] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5308.355733] Lustre: Skipped 2 previous similar messages [ 5314.066804] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5314.072226] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.080182] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5314.107412] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5314.119259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5315.553879] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5315.557644] LustreError: Skipped 4 previous similar messages [ 5321.094660] Lustre: server umount lustre-OST0001 complete [ 5328.823441] Lustre: server umount lustre-OST0000 complete [ 5336.562237] Lustre: server umount lustre-MDT0000 complete [ 5338.111501] LustreError: 236991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130682 with bad export cookie 161733465415770309 [ 5338.115271] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5338.116475] LustreError: 236991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5338.272594] Lustre: server umount lustre-MDT0001 complete [ 5342.445598] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 5343.641994] Key type lgssc unregistered [ 5343.798731] LNet: 241392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5343.802261] LNetError: 241392:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5343.812440] LNet: Removed LNI 192.168.202.119@tcp [ 5344.157898] Key type .llcrypt unregistered [ 5344.159449] Key type ._llcrypt unregistered [ 5353.891739] Key type ._llcrypt registered [ 5353.893330] Key type .llcrypt registered [ 5353.940886] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5354.284159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5354.298465] alg: No test for adler32 (adler32-zlib) [ 5355.173277] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 5355.275167] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 5356.864723] Key type lgssc registered [ 5357.313044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5358.676816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5361.262698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5361.275035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5362.483877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5364.029370] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5367.312362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5367.341259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5367.387055] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5367.401303] Lustre: 242809:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5367.406104] Lustre: Permanently deactivating lustre-MDT0001 [ 5367.412663] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5367.538684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5369.032638] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5371.061496] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5372.398174] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5372.401250] LustreError: 242850:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5372.405936] LustreError: 242850:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5372.410412] LustreError: 242850:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 1, failed: rc = -108 [ 5372.476810] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5375.058652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5375.189665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5377.251909] LustreError: 244718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.260324] LustreError: 244718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5377.564897] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5379.862153] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5382.627299] LustreError: 244436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5382.689151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5382.788920] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5385.017751] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5385.970354] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5387.091484] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5388.009926] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5388.010488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5392.785353] Lustre: Permanently reactivating lustre-MDT0001 [ 5392.793133] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5392.797071] Lustre: Skipped 2 previous similar messages [ 5393.237691] Lustre: lustre-MDT0001: Client 972aee41-95ce-4c42-901e-46bea12b560e (at 192.168.202.19@tcp) reconnecting [ 5395.368512] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5396.039536] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5398.002260] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5398.090496] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5399.609163] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5399.700943] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5400.693788] Lustre: Permanently deactivating lustre-MDT0001 [ 5400.701028] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5400.704483] Lustre: Skipped 2 previous similar messages [ 5402.951581] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5408.736899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5408.739668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5408.744896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5410.786726] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5410.787253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5410.791910] Lustre: Skipped 2 previous similar messages [ 5410.794310] Lustre: Skipped 2 previous similar messages [ 5414.595493] LustreError: 246882:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5414.632698] Lustre: server umount lustre-MDT0000 complete [ 5415.905962] LustreError: 244454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5415.915154] LustreError: 244454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5416.071234] LustreError: 242799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130760 with bad export cookie 10125598506525110122 [ 5416.073653] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5416.075864] LustreError: 242799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5416.144249] LustreError: 247083:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5416.146994] LustreError: 247083:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5416.231138] Lustre: server umount lustre-MDT0001 complete [ 5423.790825] LustreError: 247283:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5423.793209] LustreError: 247283:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5423.809833] Lustre: server umount lustre-OST0000 complete [ 5427.887102] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:12:51 (1760130771) [ 5431.823843] LustreError: 248287:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5431.826412] LustreError: 248287:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5431.899231] Lustre: server umount lustre-OST0001 complete [ 5434.332779] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 5437.952740] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5441.672469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5444.182160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5446.411592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5448.920605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5451.419271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5451.443675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.542644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5451.546906] Lustre: Skipped 2 previous similar messages [ 5451.560314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5451.602073] Lustre: lustre-MDT0000: new disk, initializing [ 5451.644121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5451.653710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5453.215485] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5458.070958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5458.107349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5458.153271] Lustre: 250469:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5458.183914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5458.187251] Lustre: Skipped 1 previous similar message [ 5458.227760] Lustre: lustre-MDT0001: new disk, initializing [ 5458.283694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5458.288029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5459.895056] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5463.341760] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5464.770754] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5467.494756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5467.529711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5467.637336] Lustre: lustre-OST0000: new disk, initializing [ 5467.639785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5467.677229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5467.679966] Lustre: Skipped 1 previous similar message [ 5469.981111] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5473.408455] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5473.787326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5473.791339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5473.832192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5479.447743] LustreError: 250476:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5484.036487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5484.069507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5484.137887] Lustre: lustre-OST0001: new disk, initializing [ 5484.140130] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5484.173504] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5486.480344] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5489.536116] LustreError: 250476:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5489.545762] LustreError: 250476:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5490.043125] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5490.689317] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5491.208066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5491.213445] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5491.256881] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5499.640154] LustreError: 250476:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5501.415526] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5501.415859] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.417158] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5501.420528] LustreError: Skipped 1 previous similar message [ 5501.426420] Lustre: Skipped 1 previous similar message [ 5506.534147] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5506.537343] Lustre: Skipped 3 previous similar messages [ 5506.552140] LustreError: 253964:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5506.555356] LustreError: 253964:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5506.589646] Lustre: server umount lustre-OST0001 complete [ 5511.653436] LustreError: 252200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5511.660027] LustreError: 252200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5516.772123] LustreError: 252200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5516.778940] LustreError: 252200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5519.843043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5519.843313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.847210] LustreError: Skipped 1 previous similar message [ 5519.854193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5519.857973] Lustre: Skipped 1 previous similar message [ 5525.243373] LustreError: 254187:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5525.250660] LustreError: 254187:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5525.352244] Lustre: server umount lustre-OST0000 complete [ 5527.011422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5527.017076] Lustre: Skipped 1 previous similar message [ 5532.130912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.133615] Lustre: Skipped 5 previous similar messages [ 5541.344137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5541.386142] LustreError: 254388:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5541.389378] LustreError: 254388:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5541.423534] Lustre: server umount lustre-MDT0000 complete [ 5542.369590] LustreError: 251428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.375944] LustreError: 251428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5542.942472] LustreError: 250459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130886 with bad export cookie 10125598506525112719 [ 5542.946620] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5542.947063] LustreError: 250459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5557.216157] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5557.344212] Lustre: server umount lustre-MDT0001 complete [ 5561.364750] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 5562.570866] Key type lgssc unregistered [ 5562.716385] LNet: 255126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5562.719205] LNetError: 255126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5562.728369] LNet: Removed LNI 192.168.202.119@tcp [ 5563.096223] Key type .llcrypt unregistered [ 5563.097872] Key type ._llcrypt unregistered [ 5575.502347] Key type ._llcrypt registered [ 5575.503843] Key type .llcrypt registered [ 5575.549456] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 5582.052254] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5582.414862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5582.474535] alg: No test for adler32 (adler32-zlib) [ 5583.345375] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 5583.434497] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 5585.024165] Key type lgssc registered [ 5585.447594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5588.848564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5591.533416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5593.869651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5596.249911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5598.874556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5598.896907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5598.902269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.011198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5600.028294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5600.069538] Lustre: lustre-MDT0000: new disk, initializing [ 5600.107737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5600.117624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5601.677369] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5606.252145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5606.274353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5606.316203] Lustre: 259046:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5606.346796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5606.349589] Lustre: Skipped 1 previous similar message [ 5606.387949] Lustre: lustre-MDT0001: new disk, initializing [ 5606.429904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5606.446901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5606.450455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5607.964236] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5611.388741] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5612.817572] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5615.552443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5615.580369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5615.703898] Lustre: lustre-OST0000: new disk, initializing [ 5615.708064] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5615.746680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5617.930533] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5621.417397] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5622.785103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5622.789790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5622.838136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5622.981574] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5623.073701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5624.526846] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5624.616050] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5627.874332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5627.877293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.879330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5627.881286] Lustre: Skipped 1 previous similar message [ 5627.885402] Lustre: Skipped 1 previous similar message [ 5631.408094] LustreError: 261987:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5631.476148] Lustre: server umount lustre-OST0000 complete [ 5632.993263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5632.993882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5632.999150] Lustre: Skipped 1 previous similar message [ 5633.002878] Lustre: Skipped 1 previous similar message [ 5638.113771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5638.115838] Lustre: Skipped 1 previous similar message [ 5643.233208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5643.235299] Lustre: Skipped 1 previous similar message [ 5647.328197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5647.329162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5647.332928] Lustre: Skipped 1 previous similar message [ 5647.367594] LustreError: 262188:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5647.370745] LustreError: 262188:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5647.399695] Lustre: server umount lustre-MDT0000 complete [ 5648.850480] LustreError: 259037:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130992 with bad export cookie 6831034616041032283 [ 5648.853649] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5648.855433] LustreError: 259037:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5648.926503] LustreError: 262391:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5648.928791] LustreError: 262391:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5649.022757] Lustre: server umount lustre-MDT0001 complete [ 5652.294952] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 17:16:36 (1760130996) [ 5654.715720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.900780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5656.344115] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5659.483658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5659.626167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5661.138814] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5663.237101] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5664.632459] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5667.188570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5669.606553] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5671.752320] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5725.667738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5725.667743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5725.668523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5725.671181] LustreError: Skipped 1 previous similar message [ 5725.678564] Lustre: Skipped 1 previous similar message [ 5727.984352] LustreError: 265404:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5727.987327] LustreError: 265404:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5728.077951] Lustre: server umount lustre-OST0000 complete [ 5730.003284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5735.078507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5735.218331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5735.221170] Lustre: Skipped 1 previous similar message [ 5737.526961] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5739.771989] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5745.445055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.453220] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.453611] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5745.462324] Lustre: Skipped 1 previous similar message [ 5776.357788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5776.358438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.360627] LustreError: Skipped 1 previous similar message [ 5776.367107] Lustre: Skipped 1 previous similar message [ 5776.369566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5776.371595] Lustre: Skipped 1 previous similar message [ 5779.629923] LustreError: 267071:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5779.632881] LustreError: 267071:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5779.733464] Lustre: server umount lustre-OST0000 complete [ 5781.474076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.478938] Lustre: Skipped 2 previous similar messages [ 5787.355064] Lustre: server umount lustre-MDT0000 complete [ 5788.713681] LustreError: 264679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131132 with bad export cookie 6831034616041033151 [ 5788.718338] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.718511] LustreError: 264679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5788.788527] LustreError: 267476:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5788.791554] LustreError: 267476:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5788.896153] Lustre: server umount lustre-MDT0001 complete [ 5792.717565] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 5793.832573] Key type lgssc unregistered [ 5793.979610] LNet: 268007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5793.982727] LNetError: 268007:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5793.989485] LNet: Removed LNI 192.168.202.119@tcp [ 5794.300454] Key type .llcrypt unregistered [ 5794.301536] Key type ._llcrypt unregistered [ 5803.485065] Key type ._llcrypt registered [ 5803.485945] Key type .llcrypt registered [ 5803.531787] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5803.949438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5803.963588] alg: No test for adler32 (adler32-zlib) [ 5804.829889] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 5804.944631] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 5806.528229] Key type lgssc registered [ 5806.952807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5809.644626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5809.653082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5810.816124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5812.070122] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5814.874828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5814.999453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5816.337923] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5818.113611] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5819.262578] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5821.624659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5821.758756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5823.874360] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5825.873162] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5835.115730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5845.475839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5845.476288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5845.476792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5845.480769] Lustre: Skipped 1 previous similar message [ 5845.483975] LustreError: Skipped 1 previous similar message [ 5850.544585] LustreError: 272419:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5850.609210] Lustre: server umount lustre-OST0000 complete [ 5855.713319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.714050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5855.717567] Lustre: Skipped 1 previous similar message [ 5855.719835] Lustre: Skipped 1 previous similar message [ 5858.175193] LustreError: 272622:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5858.177663] LustreError: 272622:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5858.205308] Lustre: server umount lustre-MDT0000 complete [ 5859.558287] LustreError: 269362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131203 with bad export cookie 12418612164555826317 [ 5859.561325] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5859.563137] LustreError: 269362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5859.633296] LustreError: 272824:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5859.635990] LustreError: 272824:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5859.733500] Lustre: server umount lustre-MDT0001 complete [ 5863.509866] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 5864.603397] Key type lgssc unregistered [ 5864.740537] LNet: 273354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5864.744761] LNetError: 273354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5864.752404] LNet: Removed LNI 192.168.202.119@tcp [ 5865.082784] Key type .llcrypt unregistered [ 5865.084080] Key type ._llcrypt unregistered [ 5872.467442] Key type ._llcrypt registered [ 5872.468675] Key type .llcrypt registered [ 5872.508985] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5872.955862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5872.963866] alg: No test for adler32 (adler32-zlib) [ 5873.829302] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 5873.926112] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 5875.520148] Key type lgssc registered [ 5875.963176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5878.934882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5878.941880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5880.109755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.500190] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5884.482418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5884.607179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5885.952314] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5887.802042] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5889.070832] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5891.497778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5891.617660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5893.563259] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5895.387888] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5906.982288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5915.104857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5915.107754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5915.112201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5917.154323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5917.154934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5917.155621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5917.156574] Lustre: Skipped 1 previous similar message [ 5920.559131] LustreError: 276948:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5920.623397] Lustre: server umount lustre-OST0000 complete [ 5922.273816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5922.274449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5922.279216] Lustre: Skipped 1 previous similar message [ 5927.393665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5927.396425] Lustre: Skipped 2 previous similar messages [ 5928.060350] LustreError: 277150:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5928.062932] LustreError: 277150:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5928.091401] Lustre: server umount lustre-MDT0000 complete [ 5929.484238] LustreError: 274514:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131273 with bad export cookie 2793052701326573390 [ 5929.487643] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5929.488316] LustreError: 274514:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5929.557201] LustreError: 277352:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5929.560295] LustreError: 277352:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5929.653387] Lustre: server umount lustre-MDT0001 complete [ 5933.738637] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 5934.915965] Key type lgssc unregistered [ 5935.064410] LNet: 277883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5935.067504] LNetError: 277883:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5935.074441] LNet: Removed LNI 192.168.202.119@tcp [ 5935.406565] Key type .llcrypt unregistered [ 5935.408258] Key type ._llcrypt unregistered [ 5945.160723] Key type ._llcrypt registered [ 5945.162017] Key type .llcrypt registered [ 5945.206760] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5945.540426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5945.551559] alg: No test for adler32 (adler32-zlib) [ 5946.424965] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 5946.522832] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 5948.120167] Key type lgssc registered [ 5948.532103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5951.294727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5951.302097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.460828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.898507] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5957.014810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5957.163668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5958.651831] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5960.584074] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5961.863525] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5964.280354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5964.416297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.482882] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5968.492771] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5978.788626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5988.321023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5988.324051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.329240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5989.350373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5989.350523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5989.350961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.358230] Lustre: Skipped 1 previous similar message [ 5992.814292] LustreError: 282300:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5992.882643] Lustre: server umount lustre-OST0000 complete [ 5994.465992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.467750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5994.472565] Lustre: Skipped 1 previous similar message [ 5994.476194] Lustre: Skipped 1 previous similar message [ 5999.586238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5999.589114] Lustre: Skipped 1 previous similar message [ 6000.524942] LustreError: 282501:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6000.527621] LustreError: 282501:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6000.561872] Lustre: server umount lustre-MDT0000 complete [ 6002.099265] LustreError: 280880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131346 with bad export cookie 12658221094613931679 [ 6002.101605] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6002.104499] LustreError: 280880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6002.178786] LustreError: 282702:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6002.182363] LustreError: 282702:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6002.269810] Lustre: server umount lustre-MDT0001 complete [ 6006.265578] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6007.442192] Key type lgssc unregistered [ 6007.592801] LNet: 283233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6007.596214] LNetError: 283233:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6007.608796] LNet: Removed LNI 192.168.202.119@tcp [ 6007.985838] Key type .llcrypt unregistered [ 6007.987054] Key type ._llcrypt unregistered [ 6015.746754] Key type ._llcrypt registered [ 6015.748382] Key type .llcrypt registered [ 6015.794385] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6016.316721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6016.331813] alg: No test for adler32 (adler32-zlib) [ 6017.203162] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6017.289926] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6018.880159] Key type lgssc registered [ 6019.292770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6022.247528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6022.255935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6023.449748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6024.817917] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6027.746054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6027.861666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6029.234663] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6031.225766] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6032.548354] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6034.924549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6035.034755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6037.020256] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6038.931449] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6048.359053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6053.859601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6053.860975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6053.864461] Lustre: Skipped 1 previous similar message [ 6053.864793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6053.867237] Lustre: Skipped 1 previous similar message [ 6055.279071] LustreError: 286862:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6055.343026] Lustre: server umount lustre-OST0000 complete [ 6058.464896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6058.468466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.473537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6058.812680] LustreError: 287063:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6058.814791] LustreError: 287063:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6058.840834] Lustre: server umount lustre-MDT0000 complete [ 6058.978190] LustreError: 284423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.983831] LustreError: 284423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6060.232373] LustreError: 286776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131404 with bad export cookie 14728776807896029948 [ 6060.235635] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6060.236732] LustreError: 286776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6060.308422] LustreError: 287266:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6060.310586] LustreError: 287266:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6060.397283] Lustre: server umount lustre-MDT0001 complete [ 6064.139339] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6065.343569] Key type lgssc unregistered [ 6065.487640] LNet: 287796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6065.491313] LNetError: 287796:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6065.502563] LNet: Removed LNI 192.168.202.119@tcp [ 6065.852068] Key type .llcrypt unregistered [ 6065.853437] Key type ._llcrypt unregistered [ 6069.964102] Key type ._llcrypt registered [ 6069.965517] Key type .llcrypt registered [ 6070.011300] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6076.147679] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6082.136690] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6082.504062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6082.542630] alg: No test for adler32 (adler32-zlib) [ 6083.410521] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6083.502805] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6085.088139] Key type lgssc registered [ 6085.540609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6088.920923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6091.100827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6093.181804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6095.370706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6097.707477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6097.732093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6097.737382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6098.824589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6098.837450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6098.875591] Lustre: lustre-MDT0000: new disk, initializing [ 6098.899856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.906561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6100.257086] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6104.464413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6104.490240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6104.529361] Lustre: 291278:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6104.562219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6104.564807] Lustre: Skipped 1 previous similar message [ 6104.607354] Lustre: lustre-MDT0001: new disk, initializing [ 6104.654210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6104.672291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6104.675848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6106.080763] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6109.293962] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6110.630845] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6113.126928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6113.153208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6113.257655] Lustre: lustre-OST0000: new disk, initializing [ 6113.260082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6113.262612] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6113.292325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6115.347596] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6118.392383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6118.395690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6118.407526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 6118.514419] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6119.996106] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6120.077199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6121.439581] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6123.590900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 6128.097642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6128.099911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6128.102125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6128.104599] Lustre: Skipped 1 previous similar message [ 6130.416163] LustreError: 294236:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 6130.481745] Lustre: server umount lustre-OST0000 complete [ 6133.218629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6133.219733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6133.223462] Lustre: Skipped 1 previous similar message [ 6133.227587] Lustre: Skipped 2 previous similar messages [ 6137.921349] LustreError: 294438:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6137.924304] LustreError: 294438:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6137.963399] Lustre: server umount lustre-MDT0000 complete [ 6139.298368] LustreError: 291271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131483 with bad export cookie 828206302948054527 [ 6139.301625] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6139.304384] LustreError: 291271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6139.372524] LustreError: 294640:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6139.375713] LustreError: 294640:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6139.458807] Lustre: server umount lustre-MDT0001 complete [ 6142.459879] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:24:46 (1760131486) [ 6144.782123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.930116] LustreError: 295309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6144.937033] LustreError: 295309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6144.953844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6146.288541] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6149.205944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6149.336279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6150.759236] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6152.641319] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6153.854495] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6156.135539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6158.282742] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6160.093468] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6174.689543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6174.691657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.692815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6174.695230] Lustre: Skipped 1 previous similar message [ 6179.565107] LustreError: 297578:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6179.567838] LustreError: 297578:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6179.626328] Lustre: server umount lustre-OST0000 complete [ 6183.911325] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6183.915044] LustreError: Skipped 1 previous similar message [ 6183.917281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.923667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6183.926369] Lustre: Skipped 1 previous similar message [ 6187.133479] LustreError: 297780:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6187.136358] LustreError: 297780:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6187.164405] Lustre: server umount lustre-MDT0000 complete [ 6188.469166] LustreError: 295290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131532 with bad export cookie 828206302948055409 [ 6188.470961] LustreError: 296029:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6188.472404] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.475297] LustreError: 295290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6188.622495] Lustre: server umount lustre-MDT0001 complete [ 6192.266740] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6193.322140] Key type lgssc unregistered [ 6193.455422] LNet: 298513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6193.458071] LNetError: 298513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6193.464588] LNet: Removed LNI 192.168.202.119@tcp [ 6193.781372] Key type .llcrypt unregistered [ 6193.782314] Key type ._llcrypt unregistered [ 6202.515927] Key type ._llcrypt registered [ 6202.516924] Key type .llcrypt registered [ 6202.555917] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6208.428550] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6208.826539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6208.833991] alg: No test for adler32 (adler32-zlib) [ 6209.697474] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6209.791620] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6211.376115] Key type lgssc registered [ 6211.781728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6215.049532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6217.623945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6219.717270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6221.984940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6223.710660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6225.407318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6227.722033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6227.743777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6227.749916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.834116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6228.845460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6228.880572] Lustre: lustre-MDT0000: new disk, initializing [ 6228.907694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6228.913730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6230.271614] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6234.594120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6234.620664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6234.657693] Lustre: 302290:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6234.682516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6234.685076] Lustre: Skipped 1 previous similar message [ 6234.722854] Lustre: lustre-MDT0001: new disk, initializing [ 6234.763891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6234.779376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6234.783402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6236.230164] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6239.419905] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6240.750025] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6243.123966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6243.150557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6243.246693] Lustre: lustre-OST2710: new disk, initializing [ 6243.248794] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6243.277616] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6245.273722] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6247.037540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6247.041631] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6247.074361] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6248.435464] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6250.786632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6250.821430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6250.884322] Lustre: lustre-OST03e8: new disk, initializing [ 6250.886994] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6250.920458] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6252.985465] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6254.952164] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6255.113302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6255.117138] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6255.146207] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6257.631319] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6257.705601] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6259.035564] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6259.129115] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6260.501552] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6260.582367] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6261.933577] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6262.007912] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6265.314261] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.316224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.317950] Lustre: Skipped 3 previous similar messages [ 6265.319756] Lustre: Skipped 2 previous similar messages [ 6269.051893] LustreError: 306629:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6269.077834] Lustre: server umount lustre-MDT0000 complete [ 6270.279230] LustreError: 302281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131614 with bad export cookie 10535243043862646221 [ 6270.282401] LustreError: 302281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6270.282727] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6270.345501] LustreError: 306831:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6270.347678] LustreError: 306831:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6270.418546] Lustre: server umount lustre-MDT0001 complete [ 6281.842099] LustreError: 307032:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6281.843973] LustreError: 307032:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6281.859879] Lustre: server umount lustre-OST2710 complete [ 6293.297632] LustreError: 307235:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6293.300026] LustreError: 307235:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6293.365838] Lustre: server umount lustre-OST03e8 complete [ 6299.117289] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6301.914638] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6305.332485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6307.603953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6309.590028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6311.739036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6313.923814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6313.941389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.022391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6314.033412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6314.067682] Lustre: lustre-MDT0000: new disk, initializing [ 6314.091170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.098747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6315.417367] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6319.524292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6319.539890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6319.568462] Lustre: 310024:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6319.659070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6320.863775] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6323.899545] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6325.049221] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6327.179422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6327.202410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6327.289825] Lustre: lustre-OST0000: new disk, initializing [ 6327.291066] Lustre: Skipped 1 previous similar message [ 6327.292733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6327.295664] Lustre: Skipped 2 previous similar messages [ 6327.320621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6327.322527] Lustre: Skipped 1 previous similar message [ 6328.248814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6328.251335] Lustre: Skipped 1 previous similar message [ 6328.253351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6328.279519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6329.206810] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6332.351852] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6333.666982] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6333.742171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6335.062357] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6335.138088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6338.530847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6338.531029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6338.531826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6338.531830] Lustre: Skipped 1 previous similar message [ 6338.533052] LustreError: Skipped 1 previous similar message [ 6338.538960] Lustre: Skipped 1 previous similar message [ 6341.869480] LustreError: 312961:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6341.871762] LustreError: 312961:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6341.928254] Lustre: server umount lustre-OST0000 complete [ 6343.649123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.649627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6343.652992] Lustre: Skipped 1 previous similar message [ 6343.654436] Lustre: Skipped 1 previous similar message [ 6348.768992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6348.771635] Lustre: Skipped 2 previous similar messages [ 6349.267267] Lustre: server umount lustre-MDT0000 complete [ 6350.448955] LustreError: 310016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131694 with bad export cookie 10535243043862647901 [ 6350.450787] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6350.453110] LustreError: 310016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6350.516232] LustreError: 313363:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6350.518023] LustreError: 313363:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6350.593286] Lustre: server umount lustre-MDT0001 complete [ 6353.318414] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:28:17 (1760131697) [ 6353.908258] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6354.469724] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:28:18 (1760131698) [ 6355.536513] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6358.258693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6359.677622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6361.046476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6362.407147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6362.808981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.372868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6365.498686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6365.512489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6365.590417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6367.147029] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6370.441610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6370.479912] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6370.493062] Lustre: 315488:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6372.035093] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6374.133694] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6375.618301] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6378.304579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6378.333846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6378.420675] Lustre: Denying initial registration attempt from nid 192.168.202.119@tcp, specified as failover [ 6378.424671] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6378.429238] LustreError: 317105:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 6378.432251] LustreError: 317105:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6378.435421] LustreError: 317105:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6378.487952] Lustre: server umount lustre-OST0000 complete [ 6378.489272] LustreError: 317105:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6382.053562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6382.059927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6387.264163] LustreError: 317472:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6387.267735] LustreError: 317472:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6387.295751] Lustre: server umount lustre-MDT0000 complete [ 6388.564365] LustreError: 317473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131732 with bad export cookie 10535243043862648769 [ 6388.567140] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6388.567518] LustreError: 317473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6388.704560] Lustre: server umount lustre-MDT0001 complete [ 6392.667131] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6393.848417] Key type lgssc unregistered [ 6393.984394] LNet: 318205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6393.987258] LNetError: 318205:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6393.995517] LNet: Removed LNI 192.168.202.119@tcp [ 6394.330352] Key type .llcrypt unregistered [ 6394.332086] Key type ._llcrypt unregistered [ 6398.080305] Key type ._llcrypt registered [ 6398.081736] Key type .llcrypt registered [ 6398.122774] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6398.490031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6398.536075] alg: No test for adler32 (adler32-zlib) [ 6399.433175] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6399.529456] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6401.120258] Key type lgssc registered [ 6401.522772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6404.159178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6405.836408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6407.363541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6408.805337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6409.224892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6416.511818] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6419.844553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6419.855965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.977546] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6420.987866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6421.056913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6422.451667] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6425.564429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6425.610991] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6425.624266] Lustre: 320937:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6425.738987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6427.244507] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6429.292569] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6430.685632] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6433.227570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6433.258117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6433.335566] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6433.418297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.702608] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6437.817275] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6440.417548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6440.420549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.422176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6440.425348] Lustre: Skipped 1 previous similar message [ 6440.430012] Lustre: Skipped 1 previous similar message [ 6444.655224] LustreError: 323222:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6444.726130] Lustre: server umount lustre-OST0000 complete [ 6446.561493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6446.565291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.571256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6450.657970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6450.658829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.777260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6455.780029] Lustre: Skipped 2 previous similar messages [ 6460.384130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6460.421216] LustreError: 323423:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6460.423915] LustreError: 323423:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6460.447757] Lustre: server umount lustre-MDT0000 complete [ 6461.929368] LustreError: 321665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131805 with bad export cookie 13662696046519012272 [ 6461.933741] LustreError: 321665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6461.934036] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6462.006821] LustreError: 323629:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6462.009266] LustreError: 323629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6462.104346] Lustre: server umount lustre-MDT0001 complete [ 6466.307474] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6467.673333] Key type lgssc unregistered [ 6467.825608] LNet: 324159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6467.829289] LNetError: 324159:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6467.836679] LNet: Removed LNI 192.168.202.119@tcp [ 6468.233157] Key type .llcrypt unregistered [ 6468.234965] Key type ._llcrypt unregistered [ 6478.924269] Key type ._llcrypt registered [ 6478.925298] Key type .llcrypt registered [ 6478.968960] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6479.295041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6479.342557] alg: No test for adler32 (adler32-zlib) [ 6480.196679] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6480.286585] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6481.864149] Key type lgssc registered [ 6482.242463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6485.187119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6485.198682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6486.378712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6487.908797] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6491.071908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6491.241679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6492.657483] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6494.427201] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6495.586353] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6497.863956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6497.978429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.886853] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6501.762799] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6516.706344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6516.707533] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.708436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6516.716989] Lustre: Skipped 2 previous similar messages [ 6517.675899] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 6517.678439] Lustre: Skipped 2 previous similar messages [ 6520.767564] LustreError: 327801:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6520.798324] Lustre: server umount lustre-MDT0000 complete [ 6521.827941] LustreError: 326874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6521.834062] LustreError: 326874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6522.144409] LustreError: 327151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131866 with bad export cookie 1458272332893069994 [ 6522.146992] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6522.148430] LustreError: 327151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6522.788236] LustreError: 326874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6522.794996] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [ 6526.945041] LustreError: 326874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6526.945349] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.958426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6528.252568] LustreError: 328003:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6528.255616] LustreError: 328003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6528.331868] Lustre: server umount lustre-MDT0001 complete [ 6530.508878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6533.251383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.424592] LustreError: 324843:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d7002d55 x1845631987506560/t0(0) o250->MGC192.168.202.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6547.533411] LustreError: 327138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.538605] LustreError: 327138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6547.566384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.591774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 6548.969879] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6552.115990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6552.234979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6553.254694] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6553.480727] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6555.104671] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6557.348762] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6558.727733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6559.850455] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6560.404878] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6562.787470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6562.787709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6562.788125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6562.788130] Lustre: Skipped 2 previous similar messages [ 6562.789571] LustreError: Skipped 1 previous similar message [ 6562.795939] Lustre: Skipped 1 previous similar message [ 6567.727052] LustreError: 330306:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6567.728707] LustreError: 330306:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6567.785455] Lustre: server umount lustre-OST0000 complete [ 6573.025438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6573.025516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.030800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6573.033524] Lustre: Skipped 2 previous similar messages [ 6575.161963] LustreError: 330508:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6575.164551] LustreError: 330508:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6575.191508] Lustre: server umount lustre-MDT0000 complete [ 6576.450178] LustreError: 330307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131920 with bad export cookie 1458272332893077148 [ 6576.453095] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6576.455543] LustreError: 330307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6576.586623] Lustre: server umount lustre-MDT0001 complete [ 6579.996284] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6580.942448] Key type lgssc unregistered [ 6581.064360] LNet: 331240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6581.067252] LNetError: 331240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6581.075413] LNet: Removed LNI 192.168.202.119@tcp [ 6581.409288] Key type .llcrypt unregistered [ 6581.411340] Key type ._llcrypt unregistered [ 6590.545810] Key type ._llcrypt registered [ 6590.546928] Key type .llcrypt registered [ 6590.589720] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6591.030890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6591.065037] alg: No test for adler32 (adler32-zlib) [ 6591.920530] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6592.030954] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6593.616214] Key type lgssc registered [ 6594.012324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6596.876227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6596.883293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.030971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.451239] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6602.562824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6602.699051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6604.132564] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6606.066955] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6607.515256] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6612.705621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.710182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6612.713190] Lustre: Skipped 1 previous similar message [ 6614.717076] LustreError: 334005:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6614.743778] Lustre: server umount lustre-MDT0000 complete [ 6615.915407] LustreError: 334007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131959 with bad export cookie 14074420760896387377 [ 6615.917891] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6615.920487] LustreError: 334007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6615.983883] LustreError: 334209:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6615.985722] LustreError: 334209:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6616.064067] Lustre: server umount lustre-MDT0001 complete [ 6620.683572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.773166] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6620.781842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6620.842740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6622.179676] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6625.148840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6625.177699] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6625.189273] Lustre: 334980:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6625.286603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6626.504616] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6628.262710] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6629.492157] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6632.037547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6632.120106] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6632.124946] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6632.128326] LustreError: 336635:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 6632.130553] LustreError: 336635:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6632.132423] LustreError: 336635:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6632.196327] Lustre: server umount lustre-OST0000 complete [ 6632.197672] LustreError: 336635:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6634.929485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6634.986435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6635.053052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6636.766091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 6637.002563] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6638.874958] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6641.451812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6641.478109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6641.503452] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6641.536752] Lustre: lustre-OST0001: new disk, initializing [ 6641.538583] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6643.660530] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6645.392804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6645.396866] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6645.432953] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6645.588935] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6650.853529] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6650.854157] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.857482] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6650.861746] Lustre: Skipped 2 previous similar messages [ 6650.864912] Lustre: Skipped 1 previous similar message [ 6652.651112] LustreError: 338754:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6652.652774] LustreError: 338754:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6652.672635] Lustre: server umount lustre-OST0001 complete [ 6655.971110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6655.971834] LustreError: 337103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6655.972469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6655.974941] Lustre: Skipped 1 previous similar message [ 6655.979509] LustreError: 337103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6661.091684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6661.092393] LustreError: 337105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6661.093561] Lustre: Skipped 2 previous similar messages [ 6661.098122] LustreError: 337105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6666.211560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6666.212073] LustreError: 338959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6666.214304] LustreError: 338959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6666.223930] Lustre: Skipped 1 previous similar message [ 6668.256113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6668.272464] LustreError: 338957:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6668.274831] LustreError: 338957:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6668.340759] Lustre: server umount lustre-OST0000 complete [ 6671.330233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6671.330271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6671.333587] LustreError: Skipped 1 previous similar message [ 6675.707631] LustreError: 339161:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6675.710503] LustreError: 339161:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6675.738094] Lustre: server umount lustre-MDT0000 complete [ 6676.449816] LustreError: 334991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6676.454588] LustreError: 334991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6676.961243] LustreError: 334970:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132020 with bad export cookie 14074420760896387923 [ 6676.963795] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6676.966370] LustreError: 334970:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6677.102305] Lustre: server umount lustre-MDT0001 complete [ 6680.934846] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6682.105096] Key type lgssc unregistered [ 6682.259359] LNet: 339892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6682.261692] LNetError: 339892:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6682.270316] LNet: Removed LNI 192.168.202.119@tcp [ 6682.599827] Key type .llcrypt unregistered [ 6682.601190] Key type ._llcrypt unregistered [ 6694.218686] Key type ._llcrypt registered [ 6694.220098] Key type .llcrypt registered [ 6694.259256] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6700.992423] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6701.494562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6701.510418] alg: No test for adler32 (adler32-zlib) [ 6702.376712] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6702.472791] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6704.064147] Key type lgssc registered [ 6704.479687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6707.898273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6710.427676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6712.798427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6715.316846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6717.964040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6717.983650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6717.990214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6719.087268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6719.100561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6719.139851] Lustre: lustre-MDT0000: new disk, initializing [ 6719.168528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6719.177277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6720.724350] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6725.356500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6725.381069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6725.414791] Lustre: 343818:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6725.451027] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6725.453073] Lustre: Skipped 1 previous similar message [ 6725.490731] Lustre: lustre-MDT0001: new disk, initializing [ 6725.525626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6725.540310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6725.544167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6726.927126] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6730.050265] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6731.370626] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6733.758664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6733.787433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6733.894948] Lustre: lustre-OST0000: new disk, initializing [ 6733.897693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6733.925525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6736.001659] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6739.205152] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6740.477867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6740.482416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6740.525189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6740.624733] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6740.696105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6741.984714] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6742.063521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6745.571904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6745.573732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.576663] LustreError: Skipped 1 previous similar message [ 6745.583825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6745.586546] Lustre: Skipped 1 previous similar message [ 6748.779801] LustreError: 346757:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6748.842599] Lustre: server umount lustre-OST0000 complete [ 6750.689098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6750.689596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6750.693567] Lustre: Skipped 2 previous similar messages [ 6755.809039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6755.810864] Lustre: Skipped 2 previous similar messages [ 6756.222140] LustreError: 346960:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6756.223887] LustreError: 346960:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6756.252072] Lustre: server umount lustre-MDT0000 complete [ 6757.510083] LustreError: 345565:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132101 with bad export cookie 7096284490902684611 [ 6757.512909] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6757.513620] LustreError: 345565:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6757.574444] LustreError: 347161:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6757.577626] LustreError: 347161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6757.660197] Lustre: server umount lustre-MDT0001 complete [ 6760.804809] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 17:35:04 (1760132104) [ 6763.005365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6765.168164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6772.936346] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6776.263497] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6779.658516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6782.017951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6784.297322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6786.659461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6789.009138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6789.037251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6789.132927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6789.148491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6789.188778] Lustre: lustre-MDT0000: new disk, initializing [ 6789.217148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6789.225412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6790.658174] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6794.981409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6795.007662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6795.040365] Lustre: 350928:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6795.064042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6795.065835] Lustre: Skipped 1 previous similar message [ 6795.097203] Lustre: lustre-MDT0001: new disk, initializing [ 6795.127849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6795.140684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6795.144349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6796.425136] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6799.507606] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6800.693184] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6802.863574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6802.888861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6804.993506] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6806.070977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6806.074631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6806.108108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6808.057384] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6809.530417] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6809.612911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6810.993883] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6811.065466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6816.226788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6816.226901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6816.228206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6816.229086] LustreError: Skipped 1 previous similar message [ 6816.233023] Lustre: Skipped 1 previous similar message [ 6817.774100] LustreError: 353865:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6817.776283] LustreError: 353865:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6817.832959] Lustre: server umount lustre-OST0000 complete [ 6820.832905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6820.835105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.839349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6820.841417] Lustre: Skipped 1 previous similar message [ 6825.086972] LustreError: 354066:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6825.089342] LustreError: 354066:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6825.118148] Lustre: server umount lustre-MDT0000 complete [ 6826.343207] LustreError: 350918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132170 with bad export cookie 7096284490902685500 [ 6826.346272] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6826.348325] LustreError: 350918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6826.491715] Lustre: server umount lustre-MDT0001 complete [ 6829.401047] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:36:13 (1760132173) [ 6832.127717] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:36:16 (1760132176) [ 6834.907479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.044517] LustreError: 355282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.049035] LustreError: 355282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6835.067228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6835.068895] Lustre: Skipped 1 previous similar message [ 6836.306223] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6839.106292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6840.609595] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6842.761347] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6844.184501] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6846.854617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6849.005745] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6850.940830] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6865.254581] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6865.989396] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6866.653918] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6867.353186] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6868.105783] Lustre: Failing over lustre-MDT0000 [ 6868.247145] LustreError: 357740:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6868.249733] LustreError: 357740:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6868.288482] Lustre: server umount lustre-MDT0000 complete [ 6869.985021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6869.988276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6869.993386] Lustre: Skipped 1 previous similar message [ 6869.995707] LustreError: 355282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6871.981773] LustreError: 355278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6871.989101] LustreError: 355278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6875.617767] LustreError: 355282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6875.623971] LustreError: 355282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6880.737604] LustreError: 355282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6880.744689] LustreError: 355282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6881.511427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6881.547329] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6881.640834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6881.643990] Lustre: Skipped 2 previous similar messages [ 6881.665975] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6882.212304] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6882.902732] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6886.885103] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6886.891826] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6886.906075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6888.000689] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6888.505143] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6890.301917] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6890.902584] Lustre: Failing over lustre-MDT0000 [ 6891.075817] LustreError: 358812:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6891.078211] LustreError: 358812:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6891.101922] Lustre: server umount lustre-MDT0000 complete [ 6892.002373] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6892.003347] LustreError: 355277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.005055] Lustre: Skipped 4 previous similar messages [ 6892.005248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6892.010528] LustreError: 355277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6904.019655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6904.066467] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6904.071424] LustreError: 355273:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 6904.192381] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6905.529205] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6907.812418] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6909.418392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6909.420993] Lustre: Skipped 2 previous similar messages [ 6909.427648] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6909.447399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6910.712491] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6911.430859] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6914.530628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6914.531154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6914.531485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6914.531490] Lustre: Skipped 2 previous similar messages [ 6914.535107] Lustre: Skipped 2 previous similar messages [ 6920.214204] Lustre: server umount lustre-MDT0000 complete [ 6921.419284] LustreError: 355998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132265 with bad export cookie 7096284490902687992 [ 6921.422441] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6921.423398] LustreError: 355998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6921.604165] Lustre: server umount lustre-MDT0001 complete [ 6932.083421] LustreError: 360248:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6932.085455] LustreError: 360248:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6932.155936] Lustre: server umount lustre-OST0000 complete [ 6937.619451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.766702] LustreError: 361075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6937.771989] LustreError: 361075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6937.790264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.793739] Lustre: Skipped 1 previous similar message [ 6939.130696] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6942.126690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6943.629908] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6945.442893] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6946.698086] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6949.201336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6951.218222] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6953.108527] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6963.686454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6972.899064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6972.901542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6972.905715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6972.908020] Lustre: Skipped 5 previous similar messages [ 6976.229574] Lustre: server umount lustre-OST0000 complete [ 6983.638799] Lustre: server umount lustre-MDT0000 complete [ 6984.162056] LustreError: 361074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.168644] LustreError: 361074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6984.956489] LustreError: 361794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132328 with bad export cookie 7096284490902688734 [ 6984.960370] LustreError: 361794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6984.961609] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6985.107945] Lustre: server umount lustre-MDT0001 complete [ 6988.746651] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6989.912898] Key type lgssc unregistered [ 6990.043526] LNet: 364274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6990.045778] LNetError: 364274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6990.053439] LNet: Removed LNI 192.168.202.119@tcp [ 6990.392783] Key type .llcrypt unregistered [ 6990.394419] Key type ._llcrypt unregistered [ 7000.100097] Key type ._llcrypt registered [ 7000.101465] Key type .llcrypt registered [ 7000.141073] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7000.439629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7000.469093] alg: No test for adler32 (adler32-zlib) [ 7001.319131] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 7001.395061] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7002.968146] Key type lgssc registered [ 7003.302228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7005.740680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7005.746655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7006.906210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7008.247416] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7011.012349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7011.131022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7012.448521] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7014.157099] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7015.386762] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7017.650895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7017.766613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7019.733409] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7021.609294] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7031.078512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7036.385974] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7036.387094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7036.390406] Lustre: Skipped 2 previous similar messages [ 7036.391905] Lustre: Skipped 1 previous similar message [ 7041.505594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.508013] Lustre: Skipped 3 previous similar messages [ 7042.172556] LustreError: 368017:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7042.198585] Lustre: server umount lustre-MDT0000 complete [ 7043.512559] LustreError: 365730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132387 with bad export cookie 4941968975681726678 [ 7043.516803] LustreError: 365730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7043.517544] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7043.595397] LustreError: 368218:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7043.597095] LustreError: 368218:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7043.706564] Lustre: server umount lustre-MDT0001 complete [ 7055.283192] LustreError: 368418:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7055.285638] LustreError: 368418:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7055.350554] Lustre: server umount lustre-OST0000 complete [ 7059.679498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7059.821517] LustreError: 369148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.827628] LustreError: 369148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7059.846449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7061.170290] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7064.088800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7064.202155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7065.522295] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7067.333103] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7068.561452] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7070.892670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7073.060590] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7074.960928] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7084.329998] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7094.756234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7094.757590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7094.760323] Lustre: Skipped 1 previous similar message [ 7094.763050] Lustre: Skipped 2 previous similar messages [ 7098.367611] LustreError: 371516:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7098.370052] LustreError: 371516:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7098.400467] Lustre: server umount lustre-MDT0000 complete [ 7099.618127] LustreError: 369127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132443 with bad export cookie 4941968975681727770 [ 7099.620689] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7099.620987] LustreError: 369127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7099.758820] Lustre: server umount lustre-MDT0001 complete [ 7110.195942] LustreError: 371918:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7110.198528] LustreError: 371918:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7110.266768] Lustre: server umount lustre-OST0000 complete [ 7115.393252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7115.540031] LustreError: 372745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7115.563889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7115.566494] Lustre: Skipped 1 previous similar message [ 7116.863679] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7119.787807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7121.309240] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7123.156583] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7124.387500] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7126.649886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7128.606601] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7130.341601] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7140.070677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7145.443967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7145.444093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7145.445071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7145.446015] LustreError: Skipped 1 previous similar message [ 7145.450565] Lustre: Skipped 2 previous similar messages [ 7148.015083] LustreError: 375011:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7148.017458] LustreError: 375011:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7148.081366] Lustre: server umount lustre-OST0000 complete [ 7150.561402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7150.561960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7150.568169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7150.569839] Lustre: Skipped 2 previous similar messages [ 7155.538903] Lustre: server umount lustre-MDT0000 complete [ 7155.681150] LustreError: 372763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.687043] LustreError: 372763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7156.820364] LustreError: 372725:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132500 with bad export cookie 4941968975681728785 [ 7156.823848] LustreError: 372725:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7156.823953] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7156.990740] Lustre: server umount lustre-MDT0001 complete [ 7160.720854] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7161.807880] Key type lgssc unregistered [ 7161.944358] LNet: 375944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7161.946754] LNetError: 375944:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7161.954307] LNet: Removed LNI 192.168.202.119@tcp [ 7162.255848] Key type .llcrypt unregistered [ 7162.256990] Key type ._llcrypt unregistered [ 7171.859555] Key type ._llcrypt registered [ 7171.860786] Key type .llcrypt registered [ 7171.894628] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7172.265073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7172.300333] alg: No test for adler32 (adler32-zlib) [ 7173.147156] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 7173.236962] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7174.816155] Key type lgssc registered [ 7175.185867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7177.821677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7177.829139] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7177.831923] LustreError: 377342:0:(osd_handler.c:8486:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7177.840099] LustreError: 377342:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7177.843172] LustreError: 377342:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7177.846477] LustreError: 377342:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7177.850167] LustreError: 377342:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7180.369943] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7180.372251] LustreError: 377739:0:(osd_handler.c:8486:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7180.381321] LustreError: 377739:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7180.383208] LustreError: 377739:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7180.385301] LustreError: 377739:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7180.389406] LustreError: 377739:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7186.897610] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7188.039765] Key type lgssc unregistered [ 7188.182408] LNet: 378723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7188.185419] LNetError: 378723:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7188.193383] LNet: Removed LNI 192.168.202.119@tcp [ 7188.496399] Key type .llcrypt unregistered [ 7188.497295] Key type ._llcrypt unregistered [ 7194.316220] Key type ._llcrypt registered [ 7194.317342] Key type .llcrypt registered [ 7194.355354] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7200.514645] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7200.938819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7200.949735] alg: No test for adler32 (adler32-zlib) [ 7201.815381] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 7201.915095] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7203.504170] Key type lgssc registered [ 7203.839531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7206.850344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7209.102929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7211.197676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7213.348288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7215.674940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7215.696946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7215.703254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7216.797656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7216.812954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7216.851963] Lustre: lustre-MDT0000: new disk, initializing [ 7216.881502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7216.888690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7218.299747] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7222.611361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7222.629647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7222.665588] Lustre: 381826:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7222.688842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7222.691331] Lustre: Skipped 1 previous similar message [ 7222.730693] Lustre: lustre-MDT0001: new disk, initializing [ 7222.765215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7222.777955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7222.781440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7224.115591] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7227.185785] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7228.373912] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7230.574968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7230.598635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7230.695204] Lustre: lustre-OST0000: new disk, initializing [ 7230.697456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7230.725330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7232.185793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7232.189102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7232.223202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7232.641392] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7235.677498] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7236.990842] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7237.065899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7238.292718] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7238.361166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7242.721450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7242.722926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7242.724419] LustreError: Skipped 1 previous similar message [ 7242.730605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7242.733723] Lustre: Skipped 1 previous similar message [ 7245.038721] LustreError: 384763:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7245.097828] Lustre: server umount lustre-OST0000 complete [ 7247.841608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7247.842249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7247.845905] Lustre: Skipped 2 previous similar messages [ 7247.849457] Lustre: Skipped 1 previous similar message [ 7252.413790] LustreError: 384964:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7252.415817] LustreError: 384964:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7252.444113] Lustre: server umount lustre-MDT0000 complete [ 7253.760432] LustreError: 381819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132597 with bad export cookie 18000450037108608445 [ 7253.762819] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7253.766522] LustreError: 381819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7253.845267] LustreError: 385165:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7253.847502] LustreError: 385165:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7253.936596] Lustre: server umount lustre-MDT0001 complete [ 7256.857847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7257.436465] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:43:21 (1760132601) [ 7259.849969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7260.024823] LustreError: 385883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7260.031818] LustreError: 385883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7260.052673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7261.332604] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7264.275301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7264.422627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7265.731830] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7267.403865] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7268.567967] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7270.649415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7272.562412] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7274.195863] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7276.407806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7276.429295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7276.476535] Lustre: lustre-OST0001: new disk, initializing [ 7276.478541] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7276.505344] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7276.507849] Lustre: Skipped 1 previous similar message [ 7277.698060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7277.700820] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7277.734369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7278.374747] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7281.433929] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7287.723886] Lustre: lustre-OST0001: Not available for connect from 192.168.202.19@tcp (stopping) [ 7288.290776] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7288.291637] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.293683] LustreError: Skipped 1 previous similar message [ 7288.297425] Lustre: Skipped 1 previous similar message [ 7289.322722] LustreError: 389080:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7289.324389] LustreError: 389080:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7289.343653] Lustre: server umount lustre-OST0001 complete [ 7293.414516] LustreError: 387478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.419196] LustreError: 387478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7298.532485] LustreError: 387479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7298.535954] LustreError: 387479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7302.624866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7302.627564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.632489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7302.634772] Lustre: Skipped 2 previous similar messages [ 7303.653964] LustreError: 388098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.658734] LustreError: 388098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7307.372169] LustreError: 389306:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7307.374144] LustreError: 389306:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7307.434409] Lustre: server umount lustre-OST0000 complete [ 7308.769651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.773037] Lustre: Skipped 1 previous similar message [ 7308.774951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7308.776867] Lustre: Skipped 3 previous similar messages [ 7314.902778] Lustre: server umount lustre-MDT0000 complete [ 7316.116255] LustreError: 385908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132660 with bad export cookie 18000450037108609313 [ 7316.120688] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7322.234448] LustreError: 389709:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7322.237143] LustreError: 389709:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7322.306799] Lustre: server umount lustre-MDT0001 complete [ 7325.944903] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7327.012499] Key type lgssc unregistered [ 7327.145317] LNet: 390240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7327.147534] LNetError: 390240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7327.156284] LNet: Removed LNI 192.168.202.119@tcp [ 7327.459583] Key type .llcrypt unregistered [ 7327.460982] Key type ._llcrypt unregistered [ 7338.420908] Key type ._llcrypt registered [ 7338.422552] Key type .llcrypt registered [ 7338.457815] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7344.434490] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7344.858174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7344.865555] alg: No test for adler32 (adler32-zlib) [ 7345.709740] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 7345.800560] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7347.376128] Key type lgssc registered [ 7347.699449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7350.549579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7352.751069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7354.675323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7356.698813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7358.943431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7358.969417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7358.974287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7360.050780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7360.060936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7360.092232] Lustre: lustre-MDT0000: new disk, initializing [ 7360.114147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7360.120551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7361.265325] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7365.497383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7365.524377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7365.561480] Lustre: 394162:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7365.591572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7365.594981] Lustre: Skipped 1 previous similar message [ 7365.639205] Lustre: lustre-MDT0001: new disk, initializing [ 7365.675919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7365.690487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7365.695156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7367.110277] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7370.225191] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7371.359419] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7373.513455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7373.540051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7373.646689] Lustre: lustre-OST0000: new disk, initializing [ 7373.648936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7373.681389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7374.063449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7374.068500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7374.113936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7375.500618] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7378.440936] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7379.768857] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7379.846173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7381.156628] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7381.247475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7383.921702] LustreError: 397098:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7383.982697] Lustre: server umount lustre-OST0000 complete [ 7386.080808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7386.082661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7386.086288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7389.665523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7389.665776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7394.785113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7394.786754] Lustre: Skipped 2 previous similar messages [ 7395.812185] LustreError: 397299:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7395.815182] LustreError: 397299:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7395.841846] Lustre: server umount lustre-MDT0000 complete [ 7397.068365] LustreError: 394153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132741 with bad export cookie 10863889654890749312 [ 7397.071605] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7397.072326] LustreError: 394153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7407.587771] LustreError: 397502:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7407.589355] LustreError: 397502:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7407.662157] Lustre: server umount lustre-MDT0001 complete [ 7410.342268] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 17:45:54 (1760132754) [ 7413.264245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7415.932279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7416.066959] Lustre: lustre-MDT0000: new disk, initializing [ 7416.100283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.412330] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7420.259386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7420.369829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7421.696213] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7423.385260] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7424.520149] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7425.194449] LustreError: 398797:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7425.196873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7425.197791] LustreError: 398797:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7425.198067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7425.202244] Lustre: Skipped 1 previous similar message [ 7425.204611] LustreError: 398797:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 7430.305097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7430.306910] Lustre: Skipped 2 previous similar messages [ 7431.358279] LustreError: 400109:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7431.361404] LustreError: 400109:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7431.389394] Lustre: server umount lustre-MDT0000 complete [ 7432.579233] LustreError: 398749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132776 with bad export cookie 10863889654890750180 [ 7432.583351] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7432.583900] LustreError: 398749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7432.719696] Lustre: server umount lustre-MDT0001 complete [ 7435.651460] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:46:19 (1760132779) [ 7437.787767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7437.941989] LustreError: 401029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.948026] LustreError: 401029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7437.962556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7439.172476] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7441.961879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7443.376897] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7445.140553] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7446.357158] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7448.588342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7450.500200] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7452.182526] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7470.483728] Lustre: Permanently deactivating lustre-OST0000 [ 7470.488130] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7476.719827] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7477.533515] LustreError: 403744:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7477.538411] LustreError: 403744:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7477.541783] LustreError: 403744:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7478.163344] LustreError: 403792:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7478.165634] LustreError: 403792:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7484.911093] LustreError: 403890:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7484.914345] LustreError: 403890:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7484.979603] Lustre: server umount lustre-OST0000 complete [ 7485.967629] LustreError: 404042:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7485.970153] LustreError: 404042:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7488.480683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7488.482504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7488.485576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7496.164239] LustreError: 404139:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7496.166905] LustreError: 404139:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7496.192598] Lustre: server umount lustre-MDT0000 complete [ 7497.437504] LustreError: 401010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132841 with bad export cookie 10863889654890750712 [ 7497.440433] LustreError: 401010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7497.440488] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7508.008954] Lustre: server umount lustre-MDT0001 complete [ 7510.783793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7512.032098] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7512.924776] LustreError: 405183:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7514.993108] Lustre: 405522:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.119@tcp [ 7515.643297] LustreError: 405619:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7515.644555] LustreError: 405619:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7515.663490] Lustre: server umount MGS complete [ 7519.411694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.570299] LustreError: 406245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7519.576226] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7519.580442] Lustre: Skipped 1 previous similar message [ 7519.602691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7519.605479] Lustre: Skipped 2 previous similar messages [ 7520.942072] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7523.976693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7524.087397] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7525.431778] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7527.147594] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7528.294086] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7528.893497] Lustre: Permanently reactivating lustre-OST0000 [ 7528.897554] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7528.899100] Lustre: Skipped 2 previous similar messages [ 7549.408131] LustreError: 407597:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7551.784169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7551.911299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7551.914089] Lustre: Skipped 1 previous similar message [ 7553.920288] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7555.781720] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7568.353070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7568.356381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7568.361904] Lustre: Skipped 1 previous similar message [ 7568.364452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7568.367207] Lustre: Skipped 3 previous similar messages [ 7573.612170] LustreError: 409436:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7573.614324] LustreError: 409436:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7573.673265] Lustre: server umount lustre-OST0000 complete [ 7575.520971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7575.523036] LustreError: Skipped 1 previous similar message [ 7579.618870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7579.621710] Lustre: Skipped 2 previous similar messages [ 7589.344172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7589.410646] Lustre: server umount lustre-MDT0000 complete [ 7589.857283] LustreError: 406244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7589.862101] LustreError: 406244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7590.744715] LustreError: 406964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132934 with bad export cookie 10863889654890751643 [ 7590.749798] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7590.750308] LustreError: 406964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7590.907575] Lustre: server umount lustre-MDT0001 complete [ 7594.772982] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7595.883693] Key type lgssc unregistered [ 7596.029413] LNet: 410375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7596.033500] LNetError: 410375:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7596.046384] LNet: Removed LNI 192.168.202.119@tcp [ 7596.417250] Key type .llcrypt unregistered [ 7596.419196] Key type ._llcrypt unregistered [ 7602.862323] Key type ._llcrypt registered [ 7602.863310] Key type .llcrypt registered [ 7602.904176] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7609.404624] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7609.751265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7609.822091] alg: No test for adler32 (adler32-zlib) [ 7610.682667] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 7610.762819] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7612.344239] Key type lgssc registered [ 7612.746233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7615.932904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7618.274295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7620.403753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7622.512737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7624.817654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7624.839576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7624.845681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7625.951603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7625.963109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7625.998402] Lustre: lustre-MDT0000: new disk, initializing [ 7626.024204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7626.033424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7627.467892] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7631.704953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7631.723576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7631.759453] Lustre: 413478:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7631.781056] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7631.783503] Lustre: Skipped 1 previous similar message [ 7631.823327] Lustre: lustre-MDT0001: new disk, initializing [ 7631.859331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7631.873319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7631.878268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7633.295835] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7636.519672] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7637.882220] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7640.320065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7640.346384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7640.461632] Lustre: lustre-OST0000: new disk, initializing [ 7640.464493] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7640.465351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7640.525927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7642.626164] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7645.679878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7645.684502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7645.700270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7645.911297] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7647.398537] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7647.473168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7648.869659] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7648.965348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7650.787504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7650.787781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.790452] LustreError: Skipped 1 previous similar message [ 7650.791811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7650.796164] Lustre: Skipped 1 previous similar message [ 7651.695164] LustreError: 416412:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7651.789044] Lustre: server umount lustre-OST0000 complete [ 7655.906054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.906645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7655.909940] Lustre: Skipped 1 previous similar message [ 7655.913117] Lustre: Skipped 2 previous similar messages [ 7659.261229] LustreError: 416618:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 7659.262840] LustreError: 416618:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7659.289932] Lustre: server umount lustre-MDT0000 complete [ 7660.611464] LustreError: 413471:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133004 with bad export cookie 16006753086357165867 [ 7660.615110] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7660.618087] LustreError: 413471:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7660.678695] LustreError: 416820:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7660.681597] LustreError: 416820:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7660.780469] Lustre: server umount lustre-MDT0001 complete [ 7663.739677] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7664.339257] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:50:08 (1760133008) [ 7666.666753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7666.863853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7668.150882] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7670.958196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7671.096945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7672.488330] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7674.435286] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7675.790578] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7678.276641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7680.450693] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7682.435531] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7683.169341] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7694.313478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7696.870146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7696.870576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7696.871811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7696.872362] LustreError: Skipped 1 previous similar message [ 7696.875511] Lustre: Skipped 1 previous similar message [ 7701.806503] LustreError: 419996:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7701.808013] LustreError: 419996:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7701.865675] Lustre: server umount lustre-OST0000 complete [ 7707.106601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7707.106955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7707.107674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7707.107678] Lustre: Skipped 1 previous similar message [ 7707.117439] Lustre: Skipped 1 previous similar message [ 7716.321516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7716.323838] Lustre: Skipped 4 previous similar messages [ 7717.344150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7717.375743] LustreError: 420198:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7717.377994] LustreError: 420198:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7717.404525] Lustre: server umount lustre-MDT0000 complete [ 7718.643022] LustreError: 417519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133062 with bad export cookie 16006753086357166707 [ 7718.646339] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7718.647497] LustreError: 417519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7718.798769] Lustre: server umount lustre-MDT0001 complete [ 7722.314519] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7723.364142] Key type lgssc unregistered [ 7723.501338] LNet: 420931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7723.503614] LNetError: 420931:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7723.509271] LNet: Removed LNI 192.168.202.119@tcp [ 7723.793805] Key type .llcrypt unregistered [ 7723.794902] Key type ._llcrypt unregistered [ 7731.000232] Key type ._llcrypt registered [ 7731.001399] Key type .llcrypt registered [ 7731.035347] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7731.147291] Key type .llcrypt unregistered [ 7731.148540] Key type ._llcrypt unregistered [ 7737.902294] Key type ._llcrypt registered [ 7737.903063] Key type .llcrypt registered [ 7737.932818] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7738.215747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7738.240084] alg: No test for adler32 (adler32-zlib) [ 7739.080948] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 7739.155026] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7740.728183] Key type lgssc registered [ 7741.157526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7743.900891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7743.910513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.084146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7746.486933] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7749.439652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7749.558398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7751.546587] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7753.453329] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7754.721628] LustreError: 423091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7754.725922] LustreError: 423091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7755.746519] LustreError: 423091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7755.751305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7755.751762] LustreError: 423091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7755.885093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7756.017620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7757.482883] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7765.984902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7765.987054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.990936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7766.499112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7766.500810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7766.505822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.671160] LustreError: 425146:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7770.739849] Lustre: server umount lustre-OST0000 complete [ 7775.202453] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7775.205831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7775.211670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7775.214404] Lustre: Skipped 1 previous similar message [ 7778.113942] LustreError: 425348:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7778.117538] LustreError: 425348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7778.149411] Lustre: server umount lustre-MDT0000 complete [ 7779.435255] LustreError: 423072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133123 with bad export cookie 14847490290923838228 [ 7779.438456] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7779.438701] LustreError: 423072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7779.506311] LustreError: 425551:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7779.507891] LustreError: 425551:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7779.585631] Lustre: server umount lustre-MDT0001 complete [ 7783.474631] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7784.687225] Key type lgssc unregistered [ 7784.828431] LNet: 426082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7784.831221] LNetError: 426082:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7784.843342] LNet: Removed LNI 192.168.202.119@tcp [ 7785.179656] Key type .llcrypt unregistered [ 7785.181231] Key type ._llcrypt unregistered [ 7794.342952] Key type ._llcrypt registered [ 7794.343775] Key type .llcrypt registered [ 7794.378067] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7794.701904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7794.709241] alg: No test for adler32 (adler32-zlib) [ 7795.578986] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 7795.676783] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7797.264104] Key type lgssc registered [ 7797.604720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7800.040828] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7800.047913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7815.520257] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7820.640232] LustreError: 427428:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [ 7847.264198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7849.070199] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7850.825573] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7853.133942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7853.328285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7853.351659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7854.603285] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7857.422403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7857.549126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7858.827216] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7860.561670] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7861.740239] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7867.874354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7867.874515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7867.878451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7867.880538] Lustre: Skipped 1 previous similar message [ 7867.883974] Lustre: Skipped 1 previous similar message [ 7870.060118] LustreError: 429709:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7870.119960] Lustre: server umount lustre-OST0000 complete [ 7872.993929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7872.994138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7872.994707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7873.004521] Lustre: Skipped 1 previous similar message [ 7877.374592] LustreError: 429910:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7877.376327] LustreError: 429910:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7877.405639] Lustre: server umount lustre-MDT0000 complete [ 7878.113021] LustreError: 428356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.117651] LustreError: 428356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7878.607940] LustreError: 429911:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133222 with bad export cookie 5530757659124519191 [ 7878.610576] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7878.612094] LustreError: 429911:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7878.669810] LustreError: 430113:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7878.672086] LustreError: 430113:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7878.754684] Lustre: server umount lustre-MDT0001 complete [ 7882.353650] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7883.422120] Key type lgssc unregistered [ 7883.562293] LNet: 430644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7883.564978] LNetError: 430644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7883.571493] LNet: Removed LNI 192.168.202.119@tcp [ 7883.862663] Key type .llcrypt unregistered [ 7883.864623] Key type ._llcrypt unregistered [ 7892.943955] Key type ._llcrypt registered [ 7892.944850] Key type .llcrypt registered [ 7892.978209] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7893.353196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7893.359733] alg: No test for adler32 (adler32-zlib) [ 7894.208542] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 7894.283560] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7895.856154] Key type lgssc registered [ 7896.188109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7898.548178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7898.554906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7899.696836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7900.911166] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7903.664035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7903.776571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7904.944122] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7906.535278] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7907.647108] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7909.759928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7909.877687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7911.695970] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7913.514541] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7913.959448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7924.197417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.199229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7924.200223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7924.201301] Lustre: Skipped 2 previous similar messages [ 7925.164302] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 7925.166746] Lustre: Skipped 2 previous similar messages [ 7926.719384] LustreError: 434267:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7926.750756] Lustre: server umount lustre-MDT0000 complete [ 7927.516055] LustreError: 432000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.19@tcp arrived at 1760133271 with bad export cookie 1963723676000706547 [ 7927.519799] LustreError: 432000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7928.138433] LustreError: 432000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133272 with bad export cookie 1963723676000705924 [ 7928.141973] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7928.143146] LustreError: 432000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7929.313291] LustreError: 432018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7929.313451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7929.313933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7929.314795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7929.320526] LustreError: 432018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7933.344757] LustreError: 432018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.345124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7933.349764] LustreError: 432018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7934.188089] LustreError: 434469:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7934.190182] LustreError: 434469:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7934.247224] Lustre: server umount lustre-OST0000 complete [ 7936.290074] LustreError: 434823:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7936.292605] LustreError: 434823:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7936.365679] Lustre: server umount lustre-MDT0001 complete [ 7939.718686] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7940.726199] Key type lgssc unregistered [ 7940.849500] LNet: 435354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7940.852088] LNetError: 435354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7940.859481] LNet: Removed LNI 192.168.202.119@tcp [ 7941.171289] Key type .llcrypt unregistered [ 7941.172541] Key type ._llcrypt unregistered [ 7949.866835] Key type ._llcrypt registered [ 7949.868023] Key type .llcrypt registered [ 7949.902071] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7950.278818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7950.302791] alg: No test for adler32 (adler32-zlib) [ 7951.161172] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 7951.257541] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7952.840195] Key type lgssc registered [ 7953.197828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7955.695056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7955.701560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.870413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7958.261804] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7961.093421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7961.215727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7962.507361] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7964.241168] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7965.401113] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7967.619575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7967.722785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7969.615453] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7971.403592] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7982.055420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7992.237590] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [ 7992.290557] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7992.290711] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.042141] LustreError: 438998:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7994.064629] Lustre: server umount lustre-MDT0001 complete [ 7997.409519] LustreError: 437452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.409854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7997.410244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7997.410249] Lustre: Skipped 1 previous similar message [ 7997.410918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7997.410920] Lustre: Skipped 2 previous similar messages [ 7997.419509] LustreError: 437452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8001.708673] LustreError: 439200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8001.710538] LustreError: 439200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8001.769461] Lustre: server umount lustre-OST0000 complete [ 8002.529964] LustreError: 437452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8002.983349] LustreError: 439401:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8002.986262] LustreError: 439401:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8003.071757] Lustre: server umount lustre-MDT0000 complete [ 8007.355335] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8008.360820] Key type lgssc unregistered [ 8008.486436] LNet: 440084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8008.488294] LNetError: 440084:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8008.494208] LNet: Removed LNI 192.168.202.119@tcp [ 8008.772758] Key type .llcrypt unregistered [ 8008.773598] Key type ._llcrypt unregistered [ 8016.501052] Key type ._llcrypt registered [ 8016.503170] Key type .llcrypt registered [ 8016.542441] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8022.289666] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8022.677248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8022.682888] alg: No test for adler32 (adler32-zlib) [ 8023.522539] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 8023.602567] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8025.176113] Key type lgssc registered [ 8025.509495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8028.206308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8030.305244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8032.214165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8034.317902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8037.725573] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8040.995180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8043.235402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8043.260056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8043.268783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.373790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8044.385631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8044.426739] Lustre: lustre-MDT0000: new disk, initializing [ 8044.454493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8044.461241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8045.863000] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8049.462431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8051.819499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8051.843687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8051.951519] Lustre: lustre-OST0000: new disk, initializing [ 8051.953503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8051.955773] Lustre: Skipped 1 previous similar message [ 8051.980514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8053.122300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8053.125471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8053.134670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8053.922045] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8057.095654] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8060.602695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8063.006047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8063.037348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8063.076644] Lustre: 444030:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8063.113317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8063.149898] Lustre: lustre-MDT0001: new disk, initializing [ 8063.206118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8063.222188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8063.225537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8064.614681] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8068.389345] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8068.476129] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8069.926856] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8072.057354] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 8075.180081] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [ 8076.258898] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8076.259124] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8076.261724] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8076.264248] Lustre: Skipped 1 previous similar message [ 8076.267048] Lustre: Skipped 1 previous similar message [ 8079.746743] LustreError: 447463:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8079.774111] Lustre: server umount lustre-MDT0001 complete [ 8086.498614] LustreError: 446269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8086.502380] LustreError: 446269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8091.618986] LustreError: 446269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8091.626514] LustreError: 446269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8094.177034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8094.179632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.183475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8096.738972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8096.739575] LustreError: 446269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8098.284099] LustreError: 447735:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 8098.286527] LustreError: 447735:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8098.367686] Lustre: server umount lustre-OST0000 complete [ 8105.980966] LustreError: 447936:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8105.982940] LustreError: 447936:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8106.089170] Lustre: server umount lustre-MDT0000 complete [ 8110.790904] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8111.949095] Key type lgssc unregistered [ 8112.081319] LNet: 448621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8112.083256] LNetError: 448621:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8112.091264] LNet: Removed LNI 192.168.202.119@tcp [ 8112.402244] Key type .llcrypt unregistered [ 8112.404279] Key type ._llcrypt unregistered [ 8125.001663] Key type ._llcrypt registered [ 8125.003014] Key type .llcrypt registered [ 8125.041299] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8125.153248] Key type .llcrypt unregistered [ 8125.154625] Key type ._llcrypt unregistered [ 8131.733845] Key type ._llcrypt registered [ 8131.735337] Key type .llcrypt registered [ 8131.774802] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8132.193689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8132.215122] alg: No test for adler32 (adler32-zlib) [ 8133.073591] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 8133.172184] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8134.752155] Key type lgssc registered [ 8135.146250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8137.909277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8140.093293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8142.050967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8144.252894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8144.276742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8144.282499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.362358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8145.372228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8145.406826] Lustre: lustre-MDT0000: new disk, initializing [ 8145.428847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.434956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8146.659576] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8150.786054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8150.809103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8150.845781] Lustre: 451936:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8150.870985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8150.872664] Lustre: Skipped 1 previous similar message [ 8150.909676] Lustre: lustre-MDT0001: new disk, initializing [ 8150.942308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8150.955913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8150.959472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8152.266766] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8155.290336] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8156.400080] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8158.508263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8158.532489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8158.632783] Lustre: lustre-OST0000: new disk, initializing [ 8158.635144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8158.668843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8160.541982] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8161.147207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8161.150394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8161.199651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8163.690661] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8166.371341] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8166.372087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8166.372667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8166.377764] Lustre: Skipped 2 previous similar messages [ 8171.489799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8171.492167] Lustre: Skipped 4 previous similar messages [ 8172.032371] LustreError: 454379:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8172.060828] Lustre: server umount lustre-MDT0000 complete [ 8173.290018] LustreError: 452717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133517 with bad export cookie 17870859066399978818 [ 8173.293534] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8173.295084] LustreError: 452717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8173.357699] LustreError: 454581:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8173.359932] LustreError: 454581:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8173.435482] Lustre: server umount lustre-MDT0001 complete [ 8184.882594] LustreError: 454782:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8184.884456] LustreError: 454782:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8184.976287] Lustre: server umount lustre-OST0000 complete [ 8188.508336] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:58:52 (1760133532) [ 8188.875157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8191.108138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8191.270502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8192.624095] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8195.367815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8195.497671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8196.699635] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8198.330912] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8199.441389] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8201.650498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8201.674958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8201.699599] LustreError: 457208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8201.707499] LustreError: 457208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8201.741150] Lustre: Found index 0 for lustre-OST0000, updating log [ 8203.528500] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8204.797785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8205.191649] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8209.888885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8209.891019] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.894939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8212.394654] LustreError: 457864:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8212.396243] LustreError: 457864:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8212.453065] Lustre: server umount lustre-OST0000 complete [ 8213.311427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8216.033587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8216.034036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8216.040623] Lustre: Skipped 1 previous similar message [ 8216.043179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8216.045848] Lustre: Skipped 4 previous similar messages [ 8222.177066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8222.178489] Lustre: Skipped 3 previous similar messages [ 8228.320145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8228.350208] LustreError: 458120:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8228.353169] LustreError: 458120:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8228.379688] Lustre: server umount lustre-MDT0000 complete [ 8229.643618] LustreError: 457778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133573 with bad export cookie 17870859066399980197 [ 8229.647828] LustreError: 457778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8229.649241] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8229.796743] Lustre: server umount lustre-MDT0001 complete [ 8232.746027] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:59:36 (1760133576) [ 8233.077842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8233.403146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8233.735151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8234.073614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8234.406372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8234.747339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8235.086889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8235.432747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8235.784526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8236.128676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8236.480180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8236.810766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8237.157108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8237.519803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8237.875891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8238.221394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8238.572564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8238.934162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8239.266206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8239.592177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8239.900616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8240.229226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8240.557143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8240.919762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8241.242952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8241.578139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8241.907620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8242.258771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8242.612041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8242.973630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8243.319415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8243.655284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8243.992402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8244.320355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8244.644527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8244.969183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8245.274520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8245.589288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8245.912138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8246.235569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8246.572143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8246.890387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8247.223329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8247.570761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8247.903531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8248.249216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8248.569576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8248.915176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8249.258637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8249.573170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8249.899578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8252.441341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8252.603047] LustreError: 461703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8252.631116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8252.634363] Lustre: Skipped 1 previous similar message [ 8253.881568] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8256.764966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8258.333392] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8260.111318] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8261.328882] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8263.614501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8263.643402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8263.737923] Lustre: Found index 0 for lustre-OST0000, updating log [ 8265.723307] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8267.478485] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8273.381302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8273.383591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8273.386860] Lustre: Skipped 1 previous similar message [ 8273.388763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8273.390385] Lustre: Skipped 2 previous similar messages [ 8274.223088] LustreError: 464019:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8274.225627] LustreError: 464019:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8274.294640] Lustre: server umount lustre-OST0000 complete [ 8281.629745] Lustre: server umount lustre-MDT0000 complete [ 8281.636565] LustreError: 462415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8281.641844] LustreError: 462415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8282.911711] LustreError: 463380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133626 with bad export cookie 17870859066399981261 [ 8282.916294] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8283.078081] Lustre: server umount lustre-MDT0001 complete [ 8284.018863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8289.373705] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8292.820770] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8296.314508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8298.503372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8300.555795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8302.667043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8304.900531] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:00:48 (1760133648) [ 8337.314645] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8338.320432] Key type lgssc unregistered [ 8338.454453] LNet: 468016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8338.456705] LNetError: 468016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8467.495533] LNet: Removed LNI 192.168.202.119@tcp [ 8467.808536] Key type .llcrypt unregistered [ 8467.809494] Key type ._llcrypt unregistered [ 8474.115503] Key type ._llcrypt registered [ 8474.116458] Key type .llcrypt registered [ 8474.151543] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8474.460030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8474.466543] alg: No test for adler32 (adler32-zlib) [ 8475.312319] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 8475.393981] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8476.968162] Key type lgssc registered [ 8477.298513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8478.616750] LNet: Added LNI 192.168.202.119@tcp100 [8/256/0/180] [ 8479.085853] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 8484.748117] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8485.905521] Key type lgssc unregistered [ 8486.075597] LNet: 469882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8486.077926] LNetError: 469882:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8486.080189] LNet: Removed LNI 192.168.202.119@tcp [ 8486.087473] LNet: Removed LNI 192.168.202.119@tcp100 [ 8486.379443] Key type .llcrypt unregistered [ 8486.380467] Key type ._llcrypt unregistered [ 8493.741530] Key type ._llcrypt registered [ 8493.743085] Key type .llcrypt registered [ 8493.778263] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8499.679387] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8499.992601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8500.029264] alg: No test for adler32 (adler32-zlib) [ 8500.892111] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 8500.986687] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8502.576219] Key type lgssc registered [ 8502.959800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8505.922584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8508.133639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8510.235184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8512.355297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8514.541046] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:04:18 (1760133858) [ 8519.999545] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8521.090547] Key type lgssc unregistered [ 8521.237360] LNet: 473893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8521.239990] LNetError: 473893:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8521.250312] LNet: Removed LNI 192.168.202.119@tcp [ 8521.575711] Key type .llcrypt unregistered [ 8521.576853] Key type ._llcrypt unregistered [ 8527.929032] Key type ._llcrypt registered [ 8527.929917] Key type .llcrypt registered [ 8527.962334] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8528.244560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8528.254742] alg: No test for adler32 (adler32-zlib) [ 8529.106267] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 8529.188869] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8530.768150] Key type lgssc registered [ 8531.124210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8534.074317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8534.092601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8534.099379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.180948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8535.192722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8535.229027] Lustre: lustre-MDT0000: new disk, initializing [ 8535.251606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8535.258541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8536.501188] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8540.617767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8540.642978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8540.681165] Lustre: 475080:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8540.711927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8540.713432] Lustre: Skipped 1 previous similar message [ 8540.748160] Lustre: lustre-MDT0001: new disk, initializing [ 8540.786036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8540.799566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8540.806733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8542.114354] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8545.278083] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8546.573758] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8548.807820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8548.828866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8548.923413] Lustre: lustre-OST0000: new disk, initializing [ 8548.925658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8548.954630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8550.794378] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8550.841924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8550.845125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8550.889607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8553.811293] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8554.926175] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing lnet_if_list [ 8556.400820] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8556.412249] LNet: Added LNI 192.168.202.119@tcp42 [8/256/0/180] [ 8566.245207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8566.245642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8566.253774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8566.256812] Lustre: Skipped 1 previous similar message [ 8568.430168] LustreError: 478460:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8568.519653] Lustre: server umount lustre-OST0000 complete [ 8571.362042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.362717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8571.367494] Lustre: Skipped 2 previous similar messages [ 8571.372641] Lustre: Skipped 1 previous similar message [ 8576.481211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8576.483802] Lustre: Skipped 1 previous similar message [ 8581.601652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8581.603980] Lustre: Skipped 1 previous similar message [ 8584.160169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8584.216698] LustreError: 478661:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8584.219929] LustreError: 478661:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8584.263760] Lustre: server umount lustre-MDT0000 complete [ 8585.655923] LustreError: 478226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133929 with bad export cookie 5183389922802613122 [ 8585.659653] LustreError: 478226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8585.660125] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8585.730652] LustreError: 478865:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8585.732504] LustreError: 478865:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8585.842147] Lustre: server umount lustre-MDT0001 complete [ 8589.811834] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8590.951073] Key type lgssc unregistered [ 8591.112668] LNet: 479404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8591.116522] LNetError: 479404:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8591.119953] LNet: Removed LNI 192.168.202.119@tcp [ 8591.126320] LNet: Removed LNI 192.168.202.119@tcp42 [ 8591.414461] Key type .llcrypt unregistered [ 8591.416634] Key type ._llcrypt unregistered [ 8598.765525] Key type ._llcrypt registered [ 8598.766468] Key type .llcrypt registered [ 8598.801529] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8599.069674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8599.117099] alg: No test for adler32 (adler32-zlib) [ 8599.961916] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 8600.036338] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8601.608107] Key type lgssc registered [ 8601.952897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8604.479980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8604.485605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.639866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8606.944386] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8609.700421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8609.825995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8611.052119] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8612.694780] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8613.822861] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8615.933501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8616.054160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8617.908334] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8619.651974] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8638.871220] loop0: detected capacity change from 0 to 204800000 [ 8638.969711] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8638.974561] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8638.979378] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8638.982933] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8639.007494] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8639.027251] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8639.048467] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8639.083080] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8639.116449] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8639.148361] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8641.163378] loop0: detected capacity change from 0 to 204800000 [ 8641.221432] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8641.273264] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8642.967787] loop0: detected capacity change from 0 to 204800000 [ 8644.867757] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8644.886612] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8644.940868] Lustre: fs15246-OST0000: new disk, initializing [ 8644.944294] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8644.963125] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.790973] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8650.598211] loop1: detected capacity change from 0 to 204800000 [ 8652.420929] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8652.448601] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8652.481607] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8652.502845] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8652.540072] Lustre: fs15246-MDT0000: new disk, initializing [ 8652.594380] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8652.602266] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8652.639482] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8653.664828] Lustre: 485017:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000004d8d9a40 x1845634210497536/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1760134013 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8653.861356] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8661.993465] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8662.752296] LustreError: 485017:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8662.757370] LustreError: 485017:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8662.761831] LustreError: 485017:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8667.108117] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8667.515416] LustreError: 486136:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 8667.540043] Lustre: server umount fs15246-MDT0000 complete [ 8670.994129] LustreError: 486641:0:(obd_class.h:479:obd_check_dev()) Device 29 not setup [ 8670.997833] LustreError: 486641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8671.013754] Lustre: server umount fs15246-OST0000 complete [ 8675.809875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8675.814677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8675.823212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8676.833190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8676.836454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8681.261432] LustreError: 487141:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8681.263473] LustreError: 487141:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8681.344736] Lustre: server umount lustre-OST0000 complete [ 8686.561697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8686.564200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8686.568079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8686.571690] Lustre: Skipped 3 previous similar messages [ 8688.778447] LustreError: 487344:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8688.779977] LustreError: 487344:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8688.812902] Lustre: server umount lustre-MDT0000 complete [ 8690.154625] LustreError: 482242:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760134034 with bad export cookie 18000452223540996096 [ 8690.159956] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8690.316763] Lustre: server umount lustre-MDT0001 complete [ 8694.104624] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8695.284959] Key type lgssc unregistered [ 8695.433408] LNet: 488077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8695.435497] LNetError: 488077:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8695.449599] LNet: Removed LNI 192.168.202.119@tcp [ 8695.778660] Key type .llcrypt unregistered [ 8695.779745] Key type ._llcrypt unregistered [ 8703.164562] Key type ._llcrypt registered [ 8703.165703] Key type .llcrypt registered [ 8703.202212] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8703.541242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8703.573978] alg: No test for adler32 (adler32-zlib) [ 8704.435225] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 8704.525158] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8706.104142] Key type lgssc registered [ 8706.478609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8709.132413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8710.848878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8712.666931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8714.614506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8719.968719] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8723.700370] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8726.990459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8728.988813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8730.915314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8732.843567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8734.989056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8735.012965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8735.018257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.108241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8736.120076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8736.169056] Lustre: lustre-MDT0000: new disk, initializing [ 8736.196938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.203897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8737.549503] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8741.660893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8741.689681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8741.721872] Lustre: 492478:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8741.748221] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8741.749678] Lustre: Skipped 1 previous similar message [ 8741.785408] Lustre: lustre-MDT0001: new disk, initializing [ 8741.816175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8741.827571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8741.830352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8743.095161] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8746.133233] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8747.289843] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8749.515143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8749.547890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8749.657575] Lustre: lustre-OST0000: new disk, initializing [ 8749.659913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8749.697375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8750.116783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8750.119443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8750.155989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8751.599092] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8754.728593] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8756.172854] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8756.255522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8757.528365] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8757.597439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8760.292813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8760.294823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.296305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8760.298215] Lustre: Skipped 1 previous similar message [ 8764.271124] LustreError: 495416:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8764.345146] Lustre: server umount lustre-OST0000 complete [ 8767.457217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8767.459164] LustreError: Skipped 1 previous similar message [ 8767.460169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.466678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8767.468522] Lustre: Skipped 1 previous similar message [ 8770.529110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8770.529758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.708944] LustreError: 495617:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8771.710726] LustreError: 495617:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8771.741758] Lustre: server umount lustre-MDT0000 complete [ 8772.975081] LustreError: 492469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760134116 with bad export cookie 10039776497500117431 [ 8772.977894] LustreError: 492469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8772.978502] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8773.037751] LustreError: 495818:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8773.040101] LustreError: 495818:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8773.127328] Lustre: server umount lustre-MDT0001 complete [ 8780.610500] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8783.544543] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8787.367728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8790.230661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8792.216338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8794.400141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8794.955519] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8538 sec ========== 18:08:58 (1760134138) [ 8795.514061] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:08:59 (1760134139) === [ 8796.608546] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:09:00 (1760134140) === [ 8809.670109] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8810.743225] Key type lgssc unregistered [ 8810.878528] LNet: 500665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8810.882950] LNetError: 500665:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8810.894551] LNet: Removed LNI 192.168.202.119@tcp [ 8811.336881] Key type .llcrypt unregistered [ 8811.338808] Key type ._llcrypt unregistered