[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509517016 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002358] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007543] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010004] pid_max: default: 32768 minimum: 301 [ 0.011134] LSM: Security Framework initializing [ 0.012052] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.022149] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026569] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027139] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029084] *** VALIDATE tmpfs *** [ 0.030331] *** VALIDATE proc *** [ 0.031243] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036023] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040384] debug: unmapping init [mem 0xffffffff8ac59000-0xffffffff8ac60fff] [ 0.042988] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044023] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049020] ... fixed-purpose events: 3 [ 0.050009] ... event mask: 000000070000000f [ 0.051298] rcu: Hierarchical SRCU implementation. [ 0.053251] smp: Bringing up secondary CPUs ... [ 0.054481] x86: Booting SMP configuration: [ 0.055029] .... node #0, CPUs: #1 #2 #3 [ 0.059374] smp: Brought up 1 node, 4 CPUs [ 0.061021] smpboot: Max logical packages: 1 [ 0.062014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128210] node 0 deferred pages initialised in 64ms [ 0.132010] devtmpfs: initialized [ 0.133371] x86/mm: Memory block size: 128MB [ 0.135942] gcov: version magic: 0x41383552 [ 0.137150] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.138077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139346] pinctrl core: initialized pinctrl subsystem [ 0.140184] [ 0.140721] ************************************************************* [ 0.141014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142012] ** ** [ 0.143011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.144009] ** ** [ 0.145013] ** This means that this kernel is built to expose internal ** [ 0.146012] ** IOMMU data structures, which may compromise security on ** [ 0.147013] ** your system. ** [ 0.148014] ** ** [ 0.149010] ** If you see this message and you are not debugging the ** [ 0.150014] ** kernel, report this immediately to your vendor! ** [ 0.151013] ** ** [ 0.152012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153015] ************************************************************* [ 0.154759] NET: Registered protocol family 16 [ 0.155448] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.156060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.157069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.159044] cpuidle: using governor menu [ 0.160873] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.163626] PCI: Using configuration type 1 for base access [ 0.165126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.174014] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.178061] cryptd: max_cpu_qlen set to 1000 [ 0.181408] ACPI: Added _OSI(Module Device) [ 0.182014] ACPI: Added _OSI(Processor Device) [ 0.183012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.184012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.188444] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.191104] ACPI: Interpreter enabled [ 0.192057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.193012] ACPI: Using IOAPIC for interrupt routing [ 0.194119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.195415] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.204483] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.205039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.206020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.207082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.209298] acpiphp: Slot [2] registered [ 0.210127] acpiphp: Slot [5] registered [ 0.211100] acpiphp: Slot [6] registered [ 0.212110] acpiphp: Slot [7] registered [ 0.212981] acpiphp: Slot [8] registered [ 0.213110] acpiphp: Slot [9] registered [ 0.214129] acpiphp: Slot [10] registered [ 0.215169] acpiphp: Slot [3] registered [ 0.216094] acpiphp: Slot [4] registered [ 0.217024] acpiphp: Slot [11] registered [ 0.218095] acpiphp: Slot [12] registered [ 0.219086] acpiphp: Slot [13] registered [ 0.220095] acpiphp: Slot [14] registered [ 0.221075] acpiphp: Slot [15] registered [ 0.222064] acpiphp: Slot [16] registered [ 0.223071] acpiphp: Slot [17] registered [ 0.224096] acpiphp: Slot [18] registered [ 0.225083] acpiphp: Slot [19] registered [ 0.226093] acpiphp: Slot [20] registered [ 0.227097] acpiphp: Slot [21] registered [ 0.228110] acpiphp: Slot [22] registered [ 0.229093] acpiphp: Slot [23] registered [ 0.230085] acpiphp: Slot [24] registered [ 0.230933] acpiphp: Slot [25] registered [ 0.231120] acpiphp: Slot [26] registered [ 0.232095] acpiphp: Slot [27] registered [ 0.233092] acpiphp: Slot [28] registered [ 0.234115] acpiphp: Slot [29] registered [ 0.235100] acpiphp: Slot [30] registered [ 0.236101] acpiphp: Slot [31] registered [ 0.237056] PCI host bridge to bus 0000:00 [ 0.238019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.239070] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.240027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.241024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.242045] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.243030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.245318] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.248121] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.249946] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.259014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.264161] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.267019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.269016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.270012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.272462] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.273492] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.276050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.278364] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.281011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.291016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.292000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.296117] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.304016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.309019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.323023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.331934] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.338015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.343018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.358015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.364000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.368015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.373013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.386015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.393121] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.399022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.404018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.431019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.448546] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.454017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.460015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.475019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.483493] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.489017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.494015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.507017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.515843] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.519370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.521379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.523250] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.525239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.530117] iommu: Default domain type: Passthrough [ 0.533670] SCSI subsystem initialized [ 0.535129] ACPI: bus type USB registered [ 0.536123] usbcore: registered new interface driver usbfs [ 0.538068] usbcore: registered new interface driver hub [ 0.540079] usbcore: registered new device driver usb [ 0.541158] pps_core: LinuxPPS API ver. 1 registered [ 0.543012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.546060] PTP clock support registered [ 0.547148] EDAC MC: Ver: 3.0.0 [ 0.549112] PCI: Using ACPI for IRQ routing [ 0.550664] NetLabel: Initializing [ 0.551009] NetLabel: domain hash size = 128 [ 0.551938] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.554084] NetLabel: unlabeled traffic allowed by default [ 0.556119] vgaarb: loaded [ 0.557204] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.558006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.565111] clocksource: Switched to clocksource kvm-clock [ 0.663454] VFS: Disk quotas dquot_6.6.0 [ 0.665579] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.668235] *** VALIDATE ramfs *** [ 0.669213] *** VALIDATE hugetlbfs *** [ 0.670657] pnp: PnP ACPI init [ 0.672955] pnp: PnP ACPI: found 6 devices [ 0.690973] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.694083] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.695890] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.697508] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.699608] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.701757] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.704211] NET: Registered protocol family 2 [ 0.706451] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.710196] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.713232] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.727089] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.730290] TCP: Hash tables configured (established 65536 bind 65536) [ 0.733054] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.735480] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.737738] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.739732] NET: Registered protocol family 1 [ 0.742688] RPC: Registered named UNIX socket transport module. [ 0.745410] RPC: Registered udp transport module. [ 0.746615] RPC: Registered tcp transport module. [ 0.748403] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.749713] NET: Registered protocol family 44 [ 0.751764] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.752904] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.754060] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.755804] PCI: CLS 0 bytes, default 64 [ 0.757628] Unpacking initramfs... [ 2.126495] debug: unmapping init [mem 0xffff8efcfcc54000-0xffff8efcfffbffff] [ 2.130359] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.132580] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.135042] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.629221] Initialise system trusted keyrings [ 2.630838] Key type blacklist registered [ 2.632626] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.641414] zbud: loaded [ 2.644564] *** VALIDATE nfs *** [ 2.646102] *** VALIDATE nfs4 *** [ 2.647671] pstore: using deflate compression [ 2.651196] Platform Keyring initialized [ 2.767177] NET: Registered protocol family 38 [ 2.768563] Key type asymmetric registered [ 2.769679] Asymmetric key parser 'x509' registered [ 2.771017] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.773372] io scheduler mq-deadline registered [ 2.774835] io scheduler kyber registered [ 2.776875] io scheduler bfq registered [ 2.779092] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.782730] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.785915] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.788582] ACPI: Power Button [PWRF] [ 2.794347] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.801118] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.813390] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.819444] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.834880] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.863484] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.898082] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.902815] Non-volatile memory driver v1.3 [ 2.904430] Linux agpgart interface v0.103 [ 2.933831] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.935958] vda: detected capacity change from 0 to 74706944 [ 2.955890] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.958874] vdb: detected capacity change from 0 to 1073741824 [ 2.974346] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.976959] vdc: detected capacity change from 0 to 2621440000 [ 2.991912] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.994110] vdd: detected capacity change from 0 to 2621440000 [ 3.020451] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.023410] vde: detected capacity change from 0 to 4294967296 [ 3.040107] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.043478] vdf: detected capacity change from 0 to 4294967296 [ 3.050084] libphy: Fixed MDIO Bus: probed [ 3.055792] usbcore: registered new interface driver usbserial_generic [ 3.058371] usbserial: USB Serial support registered for generic [ 3.060897] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.065217] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.066904] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.069532] mousedev: PS/2 mouse device common for all mice [ 3.072497] rtc_cmos 00:05: RTC can wake from S4 [ 3.074361] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.077556] rtc_cmos 00:05: registered as rtc0 [ 3.079854] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.080543] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.082751] intel_pstate: CPU model not supported [ 3.088027] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.093676] hid: raw HID events driver (C) Jiri Kosina [ 3.094841] usbcore: registered new interface driver usbhid [ 3.095882] usbhid: USB HID core driver [ 3.096754] drop_monitor: Initializing network drop monitor service [ 3.098078] Initializing XFRM netlink socket [ 3.099297] NET: Registered protocol family 10 [ 3.101762] Segment Routing with IPv6 [ 3.103159] NET: Registered protocol family 17 [ 3.105638] mpls_gso: MPLS GSO support [ 3.112873] RAS: Correctable Errors collector initialized. [ 3.114108] AVX version of gcm_enc/dec engaged. [ 3.114961] AES CTR mode by8 optimization enabled [ 3.190438] sched_clock: Marking stable (3190312124, 0)->(4114241511, -923929387) [ 3.193639] registered taskstats version 1 [ 3.195357] Loading compiled-in X.509 certificates [ 3.196976] zswap: loaded using pool lzo/zbud [ 3.221748] Key type big_key registered [ 3.231741] Key type encrypted registered [ 3.232697] ima: No TPM chip found, activating TPM-bypass! [ 3.233887] ima: Allocated hash algorithm: sha1 [ 3.234775] ima: No architecture policies found [ 3.235649] evm: Initialising EVM extended attributes: [ 3.236655] evm: security.selinux [ 3.237392] evm: security.ima [ 3.237996] evm: security.capability [ 3.238697] evm: HMAC attrs: 0x1 [ 3.240160] rtc_cmos 00:05: setting system clock to 2026-08-14 23:16:03 UTC (1786749363) [ 3.244337] debug: unmapping init [mem 0xffffffff8bc03000-0xffffffff8bdfffff] [ 3.246110] debug: unmapping init [mem 0xffffffff8a982000-0xffffffff8ac58fff] [ 3.249118] Write protecting the kernel read-only data: 28672k [ 3.252027] debug: unmapping init [mem 0xffffffff89003000-0xffffffff891fffff] [ 3.254227] debug: unmapping init [mem 0xffffffff89914000-0xffffffff899fffff] [ 3.286126] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.293812] systemd[1]: Detected virtualization kvm. [ 3.295058] systemd[1]: Detected architecture x86-64. [ 3.297217] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.323898] systemd[1]: No hostname configured. [ 3.326387] systemd[1]: Set hostname to . [ 3.329100] random: systemd: uninitialized urandom read (16 bytes read) [ 3.332601] systemd[1]: Initializing machine ID from random generator. [ 3.371055] random: ln: uninitialized urandom read (6 bytes read) [ 3.446644] random: systemd: uninitialized urandom read (16 bytes read) [ 3.449168] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.452991] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.458458] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.098695] device-mapper: uevent: version 1.0.3 [ 4.101110] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.845071] virtio_net virtio0 ens2: renamed from eth0 [ 4.871296] random: fast init done [ 4.933575] scsi host0: ata_piix [ 4.941176] scsi host1: ata_piix [ 4.943049] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.947315] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.639754] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.742400] random: crng init done [ 9.743639] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.191961] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.446303] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.716489] SELinux: Disabled at runtime. [ 11.779309] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.787255] systemd[1]: Detected virtualization kvm. [ 11.789114] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.356939] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.360831] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.374466] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.378694] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.381169] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.393450] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.398165] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.s[ 12.556608] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS lice. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.928485] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.232472] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.269355] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.425753] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.435461] EDAC sbridge: Ver: 1.1.2 [ 14.800721] Key type dns_resolver registered [ 15.104763] NFS: Registering the id_resolver key type [ 15.106856] Key type id_resolver registered [ 15.108529] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg133-server login: [ 41.447530] hrtimer: interrupt took 17513914 ns [ 75.484629] libcfs: loading out-of-tree module taints kernel. [ 75.509149] Key type ._llcrypt registered [ 75.512437] Key type .llcrypt registered [ 75.601409] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 95.273901] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 97.034498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.050431] alg: No test for adler32 (adler32-zlib) [ 98.700567] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 99.871410] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 101.719385] Key type lgssc registered [ 103.400934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.419993] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 175.327977] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 194.260415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 194.305302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.996452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.139203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.355406] Lustre: lustre-MDT0000: new disk, initializing [ 196.509850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.535912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 202.277511] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.669911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.773823] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.831970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.836727] Lustre: Skipped 1 previous similar message [ 216.905849] Lustre: lustre-MDT0001: new disk, initializing [ 216.965406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.017314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.030231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.381960] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.295458] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 236.817112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.181697] Lustre: lustre-OST0000: new disk, initializing [ 237.185794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 237.192508] Lustre: 8461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.280037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.278417] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.296310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 245.307983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 245.422328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 260.587096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.730414] Lustre: lustre-OST0001: new disk, initializing [ 260.733762] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 260.740616] Lustre: 9532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.811633] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 266.862629] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.865518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 267.889112] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 267.943218] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.742341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 286.556812] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 293.923791] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing check_logdir /tmp/testlogs/ [ 299.818498] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing yml_node [ 306.431632] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 309.367455] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 312.648705] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 314.953403] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 19:21:13 EDT 2026 [ 333.736945] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 335.319339] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 352.737449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.740914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.759796] Lustre: Skipped 1 previous similar message [ 352.777771] Lustre: Skipped 3 previous similar messages [ 357.684885] Lustre: server umount lustre-MDT0000 complete [ 361.895323] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749722 with bad export cookie 5372569826367948787 [ 361.927091] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.978088] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.980080] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.983198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.988698] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 363.033390] Lustre: Skipped 3 previous similar messages [ 367.010670] LustreError: 7940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.011930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 367.073260] LustreError: 7940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 367.095363] Lustre: Skipped 2 previous similar messages [ 372.131204] LustreError: 7940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.133449] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.155122] LustreError: 7940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 372.164662] Lustre: Skipped 1 previous similar message [ 376.287125] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 376.736340] Lustre: server umount lustre-MDT0001 complete [ 381.712893] Lustre: server umount lustre-OST0000 complete [ 386.832550] Lustre: server umount lustre-OST0001 complete [ 395.350706] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 407.013413] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 450.085888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.363831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 450.390449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 450.492815] Lustre: lustre-MDT0000: new disk, initializing [ 450.586416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.606395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 454.553317] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 465.811853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.935915] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 466.051393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 466.064147] Lustre: Skipped 1 previous similar message [ 466.239804] Lustre: lustre-MDT0001: new disk, initializing [ 466.420780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.472962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.487576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 471.718283] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 482.369929] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 489.313634] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.748292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 497.141086] Lustre: lustre-OST0000: new disk, initializing [ 497.153173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 497.180558] Lustre: 17064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 497.317652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.520605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 498.538250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 498.673396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 504.537173] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 517.040806] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 522.363343] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 522.747415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 528.611793] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 528.918682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 534.514500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 534.521070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.522916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.535089] LustreError: Skipped 1 previous similar message [ 534.549792] Lustre: Skipped 1 previous similar message [ 544.737615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.741038] Lustre: Skipped 2 previous similar messages [ 545.253129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 545.584431] Lustre: server umount lustre-OST0000 complete [ 553.441962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 553.472610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.500689] Lustre: Skipped 1 previous similar message [ 556.426221] Lustre: server umount lustre-MDT0000 complete [ 560.098823] LustreError: 15950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.116367] LustreError: 15950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 560.455963] LustreError: 15084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749920 with bad export cookie 5372569826367950362 [ 560.465159] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.483435] LustreError: 15084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 560.896792] Lustre: server umount lustre-MDT0001 complete [ 565.279386] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:25:24 (1786749924) [ 568.176501] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 570.016213] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:25:28 (1786749928) [ 576.653958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.039856] LustreError: 19475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.126847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 581.666906] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 587.234154] LustreError: 19476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.261426] LustreError: 19476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 590.031123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.684930] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 604.182915] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 610.107549] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 617.618738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 625.002370] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 633.386291] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 646.117770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 646.128445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.141505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 646.150878] Lustre: Skipped 2 previous similar messages [ 646.183872] Lustre: Skipped 5 previous similar messages [ 650.888576] Lustre: server umount lustre-OST0000 complete [ 656.356556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.151510] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 669.480137] Lustre: server umount lustre-MDT0000 complete [ 671.713622] LustreError: 19471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.743163] LustreError: 19471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 674.072532] LustreError: 19455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750034 with bad export cookie 5372569826367951286 [ 674.086437] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 674.391582] Lustre: server umount lustre-MDT0001 complete [ 686.512314] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 689.791805] Key type lgssc unregistered [ 690.144504] LNet: 23018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.155897] LNetError: 23018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.188717] LNet: Removed LNI 192.168.201.133@tcp [ 691.081467] Key type .llcrypt unregistered [ 691.093877] Key type ._llcrypt unregistered [ 714.780679] Key type ._llcrypt registered [ 714.786619] Key type .llcrypt registered [ 714.963642] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 716.156847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 716.397197] alg: No test for adler32 (adler32-zlib) [ 717.488202] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 717.718733] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 719.419654] Key type lgssc registered [ 720.483176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 729.358391] loop: module loaded [ 729.458255] loop0: detected capacity change from 0 to 204800000 [ 729.809422] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 729.818272] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 730.334519] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.272143] loop0: detected capacity change from 0 to 204800000 [ 737.490746] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.504956] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.513105] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 743.214088] loop0: detected capacity change from 0 to 204800000 [ 749.922267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 749.964253] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.366650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 751.460849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 751.651387] Lustre: lustre-MDT0000: new disk, initializing [ 751.772229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.805544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 751.859958] Lustre: 25228:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 249 < left 263, rollback = 2 [ 751.873743] Lustre: 25228:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 751.888917] Lustre: 25228:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/263/0 [ 751.897487] Lustre: 25228:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/2 [ 751.905398] Lustre: 25228:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/5, delete: 0/0/0 [ 751.916767] Lustre: 25228:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 756.552796] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 767.557538] loop1: detected capacity change from 0 to 204800000 [ 774.285333] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 774.773844] Lustre: lustre-OST07c6: new disk, initializing [ 774.786420] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 774.792822] Lustre: Skipped 1 previous similar message [ 774.797603] Lustre: 26208:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 774.911774] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 783.012084] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 783.498784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 783.519987] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 783.560786] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 789.938338] Lustre: 25194:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 789.956564] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 789.962028] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 789.969672] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 789.982671] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 789.990385] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 807.963896] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 807.978071] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 807.988122] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 807.996880] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 808.006923] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 808.012756] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 813.037751] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 813.046192] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 813.052591] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 813.058852] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.063993] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 813.068924] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.076235] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 813.082936] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.087819] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 813.092951] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.098069] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 813.104847] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.510826] Lustre: 26814:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 821.518310] Lustre: 26814:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 821.522796] Lustre: 26814:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 821.528586] Lustre: 26814:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.533651] Lustre: 26814:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 821.541349] Lustre: 26814:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.545906] Lustre: 26814:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 821.551558] Lustre: 26814:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.558685] Lustre: 26814:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 821.571774] Lustre: 26814:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.575918] Lustre: 26814:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 821.584574] Lustre: 26814:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 829.870847] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 829.877241] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 829.882467] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 829.887782] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 829.892861] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 829.898503] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 829.902884] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 829.908226] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 829.913174] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 829.918195] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 829.927670] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 829.934616] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 880.611826] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.639290] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 885.750283] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 886.362074] Lustre: server umount lustre-OST07c6 complete [ 897.494597] Lustre: server umount lustre-MDT0000 complete [ 917.168989] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 919.877188] Key type lgssc unregistered [ 920.130344] LNet: 28768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.134851] LNetError: 28768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.147596] LNet: Removed LNI 192.168.201.133@tcp [ 921.094388] Key type .llcrypt unregistered [ 921.096395] Key type ._llcrypt unregistered [ 941.361737] Key type ._llcrypt registered [ 941.367338] Key type .llcrypt registered [ 941.557808] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 943.194665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.257498] alg: No test for adler32 (adler32-zlib) [ 944.503867] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 944.810222] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 946.607420] Key type lgssc registered [ 948.045740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.331273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.354835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.847440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.745923] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 969.193453] LustreError: 30039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.217046] LustreError: 30039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 974.304233] LustreError: 30040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.549347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.995203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.474748] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 989.306521] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 997.149976] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1005.244415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1005.618880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.584148] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1022.802720] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1029.178868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1044.452232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.462841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.486299] Lustre: Skipped 1 previous similar message [ 1044.499585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1046.523790] Lustre: server umount lustre-OST0000 complete [ 1052.640427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1052.659470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.677667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1052.684072] Lustre: Skipped 1 previous similar message [ 1054.703542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1054.705546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.217685] Lustre: server umount lustre-MDT0000 complete [ 1059.809536] LustreError: 30039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.824420] LustreError: 30039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1061.731608] LustreError: 30020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750421 with bad export cookie 5825959644312123193 [ 1061.742445] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.755838] LustreError: 30020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1062.512059] Lustre: server umount lustre-MDT0001 complete [ 1076.398666] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1079.880226] Key type lgssc unregistered [ 1080.236496] LNet: 33786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.255025] LNetError: 33786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.278685] LNet: Removed LNI 192.168.201.133@tcp [ 1081.332491] Key type .llcrypt unregistered [ 1081.334225] Key type ._llcrypt unregistered [ 1102.679907] Key type ._llcrypt registered [ 1102.695603] Key type .llcrypt registered [ 1102.960703] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1104.537825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.561375] alg: No test for adler32 (adler32-zlib) [ 1105.618168] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 1105.997145] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1107.815119] Key type lgssc registered [ 1108.920224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.692060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1116.741959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.629230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.277488] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1127.907057] LustreError: 35059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.919666] LustreError: 35059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1133.026323] LustreError: 35060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.480670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.786445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1138.219288] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1147.476658] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1153.766692] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1163.570054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1163.764649] LustreError: 36882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.248910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1169.404467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1173.086940] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1182.444793] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1193.963541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.965171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.965774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1193.982564] Lustre: Skipped 1 previous similar message [ 1194.010886] Lustre: Skipped 1 previous similar message [ 1199.025451] Lustre: server umount lustre-OST0000 complete [ 1204.202407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.224531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.233725] Lustre: Skipped 1 previous similar message [ 1209.313230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1209.938426] Lustre: server umount lustre-MDT0000 complete [ 1214.025952] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750574 with bad export cookie 14363740870482404111 [ 1214.041088] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1214.044587] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1214.450735] Lustre: server umount lustre-MDT0001 complete [ 1227.117255] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1230.348082] Key type lgssc unregistered [ 1230.654827] LNet: 38598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.678324] LNetError: 38598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1231.717696] LNet: Removed LNI 192.168.201.133@tcp [ 1232.492894] Key type .llcrypt unregistered [ 1232.495026] Key type ._llcrypt unregistered [ 1257.919224] Key type ._llcrypt registered [ 1257.921888] Key type .llcrypt registered [ 1258.009783] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1259.297281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1259.530236] alg: No test for adler32 (adler32-zlib) [ 1260.729352] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 1260.986072] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1262.671216] Key type lgssc registered [ 1263.839495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1272.698685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1272.722855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.284484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.475437] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1284.578497] LustreError: 40017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.597373] LustreError: 40017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1289.709044] LustreError: 40018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.470746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.837862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1295.954641] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1304.693250] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1311.557891] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1318.439694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1318.771708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.813582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1326.142782] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1335.172702] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1343.689121] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1346.536365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.539948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.552880] Lustre: Skipped 1 previous similar message [ 1346.557777] Lustre: Skipped 1 previous similar message [ 1348.773354] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1348.895811] Lustre: server umount lustre-MDT0000 complete [ 1351.662076] LustreError: 40012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.685674] LustreError: 40012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1353.247388] LustreError: 39998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750713 with bad export cookie 11868262823250737140 [ 1353.252332] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1353.265830] LustreError: 39998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1353.894218] LustreError: 40013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.923731] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (stopping) [ 1356.769096] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.782145] Lustre: Skipped 1 previous similar message [ 1358.822066] LustreError: 41473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.834637] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1358.848305] LustreError: 41473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1358.870288] Lustre: Skipped 1 previous similar message [ 1359.632428] Lustre: server umount lustre-MDT0001 complete [ 1381.671484] Lustre: server umount lustre-OST0000 complete [ 1403.757646] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1407.384907] Key type lgssc unregistered [ 1407.708794] LNet: 43889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.720235] LNetError: 43889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.735182] LNet: Removed LNI 192.168.201.133@tcp [ 1408.787602] Key type .llcrypt unregistered [ 1408.790732] Key type ._llcrypt unregistered [ 1434.708207] Key type ._llcrypt registered [ 1434.715665] Key type .llcrypt registered [ 1434.812613] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1435.818921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1436.102075] alg: No test for adler32 (adler32-zlib) [ 1437.165691] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 1437.511642] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1439.247992] Key type lgssc registered [ 1440.706558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1448.328286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1448.348932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.775044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.938389] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1459.188379] LustreError: 45306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1459.209426] LustreError: 45306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1464.290162] LustreError: 45307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.911695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1465.163507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1469.794865] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1478.043674] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1483.979837] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1491.133032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.497533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.581352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1498.819382] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1506.298411] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1519.809590] Lustre: lustre-OST0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1520.100141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.102631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1520.116666] Lustre: Skipped 1 previous similar message [ 1523.531453] Lustre: server umount lustre-OST0000 complete [ 1542.115512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1542.121912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1542.140384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1542.147126] Lustre: Skipped 2 previous similar messages [ 1545.696482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1545.697445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.089192] Lustre: server umount lustre-MDT0000 complete [ 1550.822595] LustreError: 45306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.839645] LustreError: 45306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1551.111471] LustreError: 45288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750911 with bad export cookie 16081786704985791107 [ 1551.127347] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1551.130871] LustreError: 45288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1557.912613] Lustre: server umount lustre-MDT0001 complete [ 1570.970450] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1573.416911] Key type lgssc unregistered [ 1573.733121] LNet: 49020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.738379] LNetError: 49020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.753756] LNet: Removed LNI 192.168.201.133@tcp [ 1574.402120] Key type .llcrypt unregistered [ 1574.404119] Key type ._llcrypt unregistered [ 1599.648228] Key type ._llcrypt registered [ 1599.650442] Key type .llcrypt registered [ 1599.777811] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1600.892184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1600.943851] alg: No test for adler32 (adler32-zlib) [ 1602.187807] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 1602.504677] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1604.199198] Key type lgssc registered [ 1606.823641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1617.855138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1617.890705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1619.634455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1625.116104] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1629.158694] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.169028] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1634.119183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.279618] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1634.470288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1638.136621] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1646.647090] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1653.269717] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1660.435911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.901710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.991653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1667.180362] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1675.574994] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1684.686743] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1688.812910] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1701.803891] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1419h44m19s [ 1703.860321] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1705.952193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1705.957353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1705.969432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1707.045994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1707.049325] Lustre: Skipped 4 previous similar messages [ 1709.302602] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1709.312647] Lustre: Skipped 1 previous similar message [ 1715.363264] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1715.374246] Lustre: Skipped 3 previous similar messages [ 1720.287675] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1720.448892] Lustre: server umount lustre-MDT0000 complete [ 1721.512420] LustreError: 52283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1724.356986] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1725.413698] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.445474] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1727.663892] LustreError: 51933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.270298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.471542] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1733.916796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.007973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1734.991661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1740.822110] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1744.771430] Lustre: DEBUG MARKER: Wait for df (36708) ... [ 1748.035843] Lustre: DEBUG MARKER: done [ 1754.595324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1754.618549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.671365] Lustre: Skipped 2 previous similar messages [ 1754.676247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1754.689716] Lustre: Skipped 3 previous similar messages [ 1758.552469] Lustre: server umount lustre-OST0000 complete [ 1765.861038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.867415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1765.874186] Lustre: Skipped 1 previous similar message [ 1768.840692] Lustre: server umount lustre-MDT0000 complete [ 1773.051127] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.091508] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1773.881625] LustreError: 50422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751134 with bad export cookie 3889039758447145485 [ 1773.890783] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1773.898346] LustreError: 50422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1774.528240] Lustre: server umount lustre-MDT0001 complete [ 1786.454919] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1789.955867] Key type lgssc unregistered [ 1790.275685] LNet: 55377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.280492] LNetError: 55377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.291758] LNet: Removed LNI 192.168.201.133@tcp [ 1791.208304] Key type .llcrypt unregistered [ 1791.210949] Key type ._llcrypt unregistered [ 1836.517685] Key type ._llcrypt registered [ 1836.520695] Key type .llcrypt registered [ 1836.648936] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1838.154494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.163755] alg: No test for adler32 (adler32-zlib) [ 1839.157154] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 1839.381434] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1841.063491] Key type lgssc registered [ 1842.121636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1850.735703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1850.765546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.078565] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1852.102594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1852.224745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.299503] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1866.745827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.898317] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1866.934556] Lustre: 57585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1867.185739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1870.974379] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1878.976429] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1885.623900] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1893.486641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1893.627923] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1893.804494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.617831] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1903.798463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1910.344985] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1917.808832] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1419h47m56s [ 1919.792346] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1928.267844] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1931.937069] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1932.035249] LustreError: 57593:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1952.489657] Lustre: lustre-MDT0000: Client 981b9f38-2a02-4003-86fd-393c0118c1e5 (at 192.168.201.33@tcp) reconnecting [ 1972.090399] LustreError: 57593:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1975.230151] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1977.934470] Lustre: DEBUG MARKER: done [ 1990.627040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1990.643492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.678438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1991.151720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1992.461940] Lustre: server umount lustre-OST0000 complete [ 2000.357555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2000.366925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.381769] Lustre: Skipped 1 previous similar message [ 2000.386164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2000.391424] Lustre: Skipped 2 previous similar messages [ 2001.453090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.464188] Lustre: Skipped 2 previous similar messages [ 2007.543689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.556891] Lustre: Skipped 1 previous similar message [ 2011.103551] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2011.397637] Lustre: server umount lustre-MDT0000 complete [ 2015.557144] LustreError: 57579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751375 with bad export cookie 12714045887038524557 [ 2015.561260] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.564445] LustreError: 57579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2016.191495] Lustre: server umount lustre-MDT0001 complete [ 2029.954634] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2033.292972] Key type lgssc unregistered [ 2033.765254] LNet: 61926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.789545] LNetError: 61926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.824201] LNet: Removed LNI 192.168.201.133@tcp [ 2034.919269] Key type .llcrypt unregistered [ 2034.921808] Key type ._llcrypt unregistered [ 2090.983756] Key type ._llcrypt registered [ 2090.986964] Key type .llcrypt registered [ 2091.085533] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2092.397574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2092.631431] alg: No test for adler32 (adler32-zlib) [ 2093.747198] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 2094.052575] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2095.760084] Key type lgssc registered [ 2097.286751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2100.260702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2100.294927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.650599] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2101.682287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2101.977973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.120286] Lustre: Failing over lustre-MDT0000 [ 2103.359955] Lustre: server umount lustre-MDT0000 complete [ 2112.271398] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:51:10 (1786751470) [ 2119.836102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.953311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.438952] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2138.687129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.764950] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2138.796209] Lustre: 64729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2139.150206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2143.363149] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2153.239116] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2161.248939] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2171.466075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2171.950778] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2172.311543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.237625] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2182.412076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2188.388288] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2196.515954] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2208.242752] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.471091] Lustre: server umount lustre-MDT0000 complete [ 2212.845865] LustreError: 64723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751573 with bad export cookie 5226493985676701352 [ 2212.849754] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2212.856108] LustreError: 64723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2213.251975] Lustre: server umount lustre-MDT0001 complete [ 2217.735556] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2222.988460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2229.735253] Lustre: 63651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751573/real 1786751573] req@0000000090895d53 x1873544692674176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786751589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2229.776606] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.788690] Lustre: Skipped 2 previous similar messages [ 2232.800176] Lustre: 63652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751577/real 1786751577] req@00000000808bbc88 x1873544692674304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786751593 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2233.007759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.823684] Lustre: 63653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751578/real 1786751578] req@000000009ffd74c7 x1873544692674432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786751594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2237.984191] LustreError: 63650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fa703928 x1873544692675200/t0(0) o250->MGC192.168.201.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2238.385588] LustreError: 66581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.427639] LustreError: 66581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2238.575931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.776518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2239.519126] Lustre: 63653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751583/real 1786751583] req@00000000215e58a0 x1873544692674688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786751599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2240.628728] LustreError: 68506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.652379] LustreError: 68506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2244.639192] Lustre: 63652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751588/real 1786751588] req@000000004959587d x1873544692674816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786751604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2245.521951] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2246.115125] LustreError: 68505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.136825] LustreError: 68505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2248.168131] LustreError: 68504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.174602] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2248.195296] LustreError: 68504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2253.307231] LustreError: 66582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.351337] LustreError: 66582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2258.061830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.728355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2263.907198] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2264.065538] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2271.054291] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2277.808409] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2289.639434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2289.643132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.650705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.718037] Lustre: Skipped 2 previous similar messages [ 2294.756995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.763577] Lustre: Skipped 4 previous similar messages [ 2299.873624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2299.891935] Lustre: Skipped 2 previous similar messages [ 2301.407470] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2301.825707] Lustre: server umount lustre-MDT0000 complete [ 2305.003958] LustreError: 68807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.044353] LustreError: 68807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2306.642235] LustreError: 64721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751666 with bad export cookie 5226493985676703627 [ 2306.656627] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2306.665496] LustreError: 64721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2307.133400] Lustre: server umount lustre-MDT0001 complete [ 2314.613615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2326.495372] Lustre: 63653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751670/real 1786751670] req@0000000094602615 x1873544692705280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786751686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2326.513121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.525498] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.682455] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x48883c96bcf9bcee [ 2331.696749] Lustre: MGC192.168.201.133@tcp: Connection restored to 0@lo (at 0@lo) [ 2332.081033] LustreError: 66581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.172268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.232137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2335.274386] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2338.636462] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2343.392810] Lustre: 63654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751686/real 1786751686] req@00000000313b4860 x1873544692706048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786751702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2343.428676] Lustre: 63654:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 2349.229169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.686941] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2354.982725] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2363.704370] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2371.974827] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2378.748399] Lustre: server umount lustre-MDT0000 complete [ 2380.268242] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.291195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2380.302110] LustreError: 66581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.302275] Lustre: Skipped 1 previous similar message [ 2380.359859] LustreError: 66581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 2383.118135] LustreError: 64722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751743 with bad export cookie 5226493985676705006 [ 2383.129639] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.134278] LustreError: 64722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2383.644818] Lustre: server umount lustre-MDT0001 complete [ 2391.024488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2394.869503] Lustre: DEBUG MARKER: files compared the same [ 2407.888340] Lustre: server umount lustre-OST0000 complete [ 2425.468961] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2429.052984] Key type lgssc unregistered [ 2429.454155] LNet: 74476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.458526] LNetError: 74476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.476696] LNet: Removed LNI 192.168.201.133@tcp [ 2430.637477] Key type .llcrypt unregistered [ 2430.640249] Key type ._llcrypt unregistered [ 2458.615942] Key type ._llcrypt registered [ 2458.618542] Key type .llcrypt registered [ 2458.745404] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2460.128726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2460.224511] alg: No test for adler32 (adler32-zlib) [ 2461.373590] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 2461.694401] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2463.399398] Key type lgssc registered [ 2464.745831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2473.622644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2473.670317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.342472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.603358] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug +malloc all [ 2485.729357] LustreError: 75894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.741279] LustreError: 75894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2488.099240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.346754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2493.016864] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug +malloc all [ 2501.744354] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2509.677665] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2518.284487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.756720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.797017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2526.456926] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug +malloc all [ 2535.312436] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2545.631971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2545.641484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.671847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2548.703936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2548.705304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2548.720069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.673302] Lustre: server umount lustre-OST0000 complete [ 2558.945779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.949762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2558.982739] Lustre: Skipped 1 previous similar message [ 2559.009841] Lustre: Skipped 2 previous similar messages [ 2564.077679] LustreError: 76623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.109767] LustreError: 76623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2564.162775] Lustre: server umount lustre-MDT0000 complete [ 2568.973334] LustreError: 75875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751929 with bad export cookie 7112093502186781180 [ 2568.983533] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2568.986128] LustreError: 75875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2569.452589] Lustre: server umount lustre-MDT0001 complete [ 2582.639247] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2586.507942] Key type lgssc unregistered [ 2586.804964] LNet: 79432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2586.809643] LNetError: 79432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2586.834726] LNet: Removed LNI 192.168.201.133@tcp [ 2587.754877] Key type .llcrypt unregistered [ 2587.757342] Key type ._llcrypt unregistered [ 2615.525739] Key type ._llcrypt registered [ 2615.529240] Key type .llcrypt registered [ 2615.655629] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2617.139238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2617.185234] alg: No test for adler32 (adler32-zlib) [ 2618.289825] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 2618.685958] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2620.439292] Key type lgssc registered [ 2621.630743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2629.812819] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2629.823816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2645.347116] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2650.463402] LustreError: 80830:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 2677.151237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2683.517714] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2691.287273] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2704.125990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.417752] Lustre: *** cfs_fail_loc=706, val=0*** [ 2704.829306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.910134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2709.985823] LustreError: 80856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.034908] LustreError: 80856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2711.052037] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2712.040677] LustreError: 80855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.074483] LustreError: 80855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2717.154862] LustreError: 81928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.181349] LustreError: 81928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2719.634777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.090620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2724.418559] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2733.915907] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2740.339033] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2745.834490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2745.836089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.856180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2745.890401] Lustre: Skipped 1 previous similar message [ 2748.644634] Lustre: server umount lustre-OST0000 complete [ 2756.068057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.070313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2756.080241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2756.080250] LustreError: Skipped 1 previous similar message [ 2756.080972] Lustre: Skipped 1 previous similar message [ 2756.111851] Lustre: Skipped 2 previous similar messages [ 2759.674995] Lustre: server umount lustre-MDT0000 complete [ 2761.190144] LustreError: 81933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.205632] LustreError: 81933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2763.228845] LustreError: 80838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752123 with bad export cookie 9889963274756887523 [ 2763.237917] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2763.248663] LustreError: 80838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2763.487881] Lustre: server umount lustre-MDT0001 complete [ 2775.238653] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2778.850971] Key type lgssc unregistered [ 2779.066773] LNet: 84464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2779.070770] LNetError: 84464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2780.137956] LNet: Removed LNI 192.168.201.133@tcp [ 2780.964748] Key type .llcrypt unregistered [ 2780.966079] Key type ._llcrypt unregistered [ 2808.041377] Key type ._llcrypt registered [ 2808.042942] Key type .llcrypt registered [ 2808.154565] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2809.636368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2809.823562] alg: No test for adler32 (adler32-zlib) [ 2811.271818] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 2811.610497] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2813.327241] Key type lgssc registered [ 2814.679955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2823.002691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2823.024995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.084604] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2837.262700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.833280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2842.473615] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2843.108854] LustreError: 86568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.117285] LustreError: 86568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2847.211587] LustreError: 86572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.222111] LustreError: 86572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2851.676217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2852.004943] LustreError: 86568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.021645] LustreError: 86568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2852.059335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.440547] LustreError: 86571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.456111] LustreError: 86571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2858.620187] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2862.562745] LustreError: 86572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.584186] LustreError: 86572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2867.121252] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2867.132148] Lustre: MGS: Client ea342228-9a17-4253-b1bf-97a4ce355e8e (at 0@lo) reconnecting [ 2867.167755] LustreError: 86582:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2867.558353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2867.664609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2871.914034] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2888.163865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2888.168151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.179400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2888.184615] LustreError: Skipped 1 previous similar message [ 2888.238207] Lustre: Skipped 1 previous similar message [ 2891.740952] Lustre: server umount lustre-OST0000 complete [ 2898.401067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.404498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2898.413944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2898.427340] Lustre: Skipped 2 previous similar messages [ 2902.854687] Lustre: server umount lustre-MDT0000 complete [ 2903.520619] LustreError: 86572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.535154] LustreError: 86572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2907.166254] Lustre: server umount lustre-MDT0001 complete [ 2910.655721] Lustre: server umount MGS complete [ 2924.924126] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:04:43 (1786752283) [ 2958.292528] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 2968.095415] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3005.260495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.496641] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3022.326521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3022.642119] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3022.706802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3022.861829] Lustre: lustre-MDT0001: new disk, initializing [ 3022.977122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3023.010761] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3023.020115] Lustre: 93740:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3023.036508] LustreError: 93740:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3023.059912] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 3023.076670] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 3023.560515] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 3023.577852] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) Skipped 1308 previous similar messages [ 3023.589065] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 3023.597785] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 1308 previous similar messages [ 3024.560162] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 3024.585613] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) Skipped 3758 previous similar messages [ 3024.602147] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 3024.613310] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 3758 previous similar messages [ 3026.560172] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 3026.568784] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) Skipped 7514 previous similar messages [ 3026.596321] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 3026.608633] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 7551 previous similar messages [ 3029.784989] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3030.560111] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 3030.575134] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) Skipped 16463 previous similar messages [ 3030.599485] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 3030.617365] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 16426 previous similar messages [ 3038.561889] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 3038.575047] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) Skipped 28169 previous similar messages [ 3038.600146] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 3038.614592] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 28180 previous similar messages [ 3045.455040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3045.801602] Lustre: lustre-OST0000: new disk, initializing [ 3045.805402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3045.811653] Lustre: 94508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3045.889814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.141626] Lustre: 94685:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002a5d40b8 x1873545444501888/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786752428 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3054.562266] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 3054.578652] LustreError: 93742:0:(fld_handler.c:275:fld_server_lookup()) Skipped 63981 previous similar messages [ 3054.600307] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 3054.617819] LustreError: 93742:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 64020 previous similar messages [ 3056.156668] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3068.666493] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3072.992199] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3075.835130] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3075.850928] Lustre: MGS: Client 384192dc-1bde-4453-8020-10d75a4ec94c (at 0@lo) reconnecting [ 3075.925034] Lustre: 93009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3075.943045] LustreError: 93725:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3076.003241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3076.096171] Lustre: lustre-MDT0000: new disk, initializing [ 3076.304882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.378166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3077.483304] Lustre: 95578:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000c8cb58cd x1873545444509568/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786752453 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3082.591755] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3083.002539] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3084.156312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3084.167583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3086.709045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3086.730593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3086.759635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3109.862472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.886599] Lustre: Skipped 2 previous similar messages [ 3109.886620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3114.981302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3114.995888] Lustre: Skipped 2 previous similar messages [ 3116.022648] Lustre: server umount lustre-OST0000 complete [ 3125.224078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.230463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3125.238571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3125.249604] Lustre: Skipped 1 previous similar message [ 3127.302571] Lustre: server umount lustre-MDT0000 complete [ 3130.339729] LustreError: 95567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.367772] LustreError: 95567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3137.937425] Lustre: server umount lustre-MDT0001 complete [ 3142.489791] Lustre: server umount MGS complete [ 3156.636743] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:08:35 (1786752515) [ 3177.900195] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3181.528895] Key type lgssc unregistered [ 3181.943281] LNet: 98078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3181.958916] LNetError: 98078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3181.983418] LNet: Removed LNI 192.168.201.133@tcp [ 3183.212797] Key type .llcrypt unregistered [ 3183.216578] Key type ._llcrypt unregistered [ 3203.541203] Key type ._llcrypt registered [ 3203.543840] Key type .llcrypt registered [ 3203.697397] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3204.941662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3204.999836] alg: No test for adler32 (adler32-zlib) [ 3206.054911] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 3206.329719] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3208.015410] Key type lgssc registered [ 3209.408489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3218.982290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3219.003525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.585966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.019212] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3230.181440] LustreError: 99304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.206938] LustreError: 99304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3235.033758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.278460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3240.754099] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3250.007732] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3256.554623] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3266.018383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.635071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.534711] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3287.383141] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3290.225798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3305.952556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3305.955823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.962454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.969539] Lustre: Skipped 1 previous similar message [ 3305.983860] Lustre: Skipped 1 previous similar message [ 3308.128576] Lustre: server umount lustre-OST0000 complete [ 3316.192379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.201531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.203564] Lustre: Skipped 1 previous similar message [ 3318.681301] Lustre: server umount lustre-MDT0000 complete [ 3321.313763] LustreError: 100750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.330829] LustreError: 100750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3322.185693] LustreError: 99286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752682 with bad export cookie 5245831367902580736 [ 3322.198173] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.203689] LustreError: 99286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3322.589271] Lustre: server umount lustre-MDT0001 complete [ 3335.022151] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3338.178885] Key type lgssc unregistered [ 3338.420753] LNet: 102867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3338.425511] LNetError: 102867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3338.435769] LNet: Removed LNI 192.168.201.133@tcp [ 3339.155663] Key type .llcrypt unregistered [ 3339.158237] Key type ._llcrypt unregistered [ 3356.239779] Key type ._llcrypt registered [ 3356.242517] Key type .llcrypt registered [ 3356.384268] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3358.091147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3358.144574] alg: No test for adler32 (adler32-zlib) [ 3359.126178] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 3359.338706] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3360.999294] Key type lgssc registered [ 3362.356581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3368.583996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3368.589633] LustreError: 103934:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3368.669834] LustreError: 103952:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3368.673627] LustreError: 103934:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4928 [ 3368.678612] LustreError: 103934:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3368.686600] LustreError: 103934:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3368.702824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.158143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.920949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.281129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3383.216567] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3388.056940] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing lsmod [ 3390.671712] LustreError: 105158:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3390.691703] LustreError: 105159:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3390.697598] LustreError: 105158:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4978 [ 3390.707767] LustreError: 105158:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3390.713309] LustreError: 105158:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3390.721520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3391.196582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.586237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.596852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3396.601360] Lustre: Skipped 1 previous similar message [ 3396.615611] Lustre: Skipped 2 previous similar messages [ 3401.698978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3401.708691] Lustre: Skipped 2 previous similar messages [ 3403.750205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.754490] Lustre: Skipped 2 previous similar messages [ 3408.351234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3408.558156] Lustre: server umount lustre-MDT0000 complete [ 3408.865108] LustreError: 103988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.873307] LustreError: 103988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3411.785665] LustreError: 105343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752772 with bad export cookie 15731240351869790673 [ 3411.791644] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3411.797092] LustreError: 105343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3411.939331] LustreError: 104420:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003224db67 x1873546019099648/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3412.157680] Lustre: server umount lustre-MDT0001 complete [ 3425.785474] Lustre: server umount lustre-OST0000 complete [ 3436.279331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.301703] LustreError: 106223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3437.388954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3442.664710] LustreError: 106224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3443.709050] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3446.757642] LustreError: 106223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.873591] LustreError: 106224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.990514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.748967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3459.845230] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3467.981442] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3473.582395] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3480.478978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.733068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3481.817868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3488.019124] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3497.916321] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3504.619339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.619528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.640247] Lustre: Skipped 1 previous similar message [ 3504.641516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.678348] Lustre: Skipped 1 previous similar message [ 3509.735622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.754775] Lustre: Skipped 1 previous similar message [ 3510.650836] Lustre: server umount lustre-OST0000 complete [ 3514.848680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.864927] Lustre: Skipped 2 previous similar messages [ 3519.970469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.973150] Lustre: Skipped 2 previous similar messages [ 3520.484737] Lustre: server umount lustre-MDT0000 complete [ 3524.342855] LustreError: 106203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752884 with bad export cookie 15731240351869791359 [ 3524.347022] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3524.353919] LustreError: 106203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3524.739836] Lustre: server umount lustre-MDT0001 complete [ 3539.053553] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3543.831745] Key type lgssc unregistered [ 3544.211930] LNet: 109764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3544.220113] LNetError: 109764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3544.262405] LNet: Removed LNI 192.168.201.133@tcp [ 3545.712658] Key type .llcrypt unregistered [ 3545.716320] Key type ._llcrypt unregistered [ 3574.196534] Key type ._llcrypt registered [ 3574.205033] Key type .llcrypt registered [ 3574.403978] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3577.010423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3577.032752] alg: No test for adler32 (adler32-zlib) [ 3578.428119] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 3578.822179] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3580.567286] Key type lgssc registered [ 3582.984392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3592.611463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3592.636691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.233396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3598.765879] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3604.459438] LustreError: 111183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3604.478813] LustreError: 111183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3608.064223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.488407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3614.415238] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3624.446582] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3630.927111] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3639.099140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3639.279363] LustreError: 113007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.563231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3644.928499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3647.324062] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3657.011662] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3663.398400] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3667.615840] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3673.569923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3673.575619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.581371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.609500] Lustre: Skipped 1 previous similar message [ 3678.508943] Lustre: server umount lustre-OST0000 complete [ 3683.810304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.830705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.838667] Lustre: Skipped 1 previous similar message [ 3683.848791] Lustre: Skipped 1 previous similar message [ 3688.933038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3688.947753] Lustre: Skipped 2 previous similar messages [ 3689.731985] Lustre: server umount lustre-MDT0000 complete [ 3693.026277] LustreError: 111180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3693.042221] LustreError: 111180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3693.507133] LustreError: 113719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753053 with bad export cookie 698330484851340475 [ 3693.509299] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3693.518295] LustreError: 113719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3693.837410] Lustre: server umount lustre-MDT0001 complete [ 3705.794596] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3708.153373] Key type lgssc unregistered [ 3708.477845] LNet: 114829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3708.492583] LNetError: 114829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3708.507423] LNet: Removed LNI 192.168.201.133@tcp [ 3709.519321] Key type .llcrypt unregistered [ 3709.521335] Key type ._llcrypt unregistered [ 3727.373492] Key type ._llcrypt registered [ 3727.375022] Key type .llcrypt registered [ 3727.530857] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3728.642184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3728.672068] alg: No test for adler32 (adler32-zlib) [ 3730.016774] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 3730.345248] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3732.047415] Key type lgssc registered [ 3732.931655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3742.026839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3742.044706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.618304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.032674] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3752.932458] LustreError: 116052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.951607] LustreError: 116052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3757.334062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.592480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3762.963918] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3772.797125] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3778.000388] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3785.911109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3786.378421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.469297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3792.784347] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3800.552843] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3809.248280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3809.252387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3809.271203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3812.321381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3812.328049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3812.333491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3814.439925] Lustre: server umount lustre-OST0000 complete [ 3818.977535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3818.987136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3819.005621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3819.012548] Lustre: Skipped 1 previous similar message [ 3822.562630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3822.566625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.680914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3827.689036] Lustre: Skipped 2 previous similar messages [ 3832.287263] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3832.589792] Lustre: server umount lustre-MDT0000 complete [ 3836.227856] LustreError: 116032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753196 with bad export cookie 17608796221475418201 [ 3836.232473] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.240695] LustreError: 116032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3836.598786] Lustre: server umount lustre-MDT0001 complete [ 3847.754384] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3850.904440] Key type lgssc unregistered [ 3851.165887] LNet: 119595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3851.174340] LNetError: 119595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3851.202107] LNet: Removed LNI 192.168.201.133@tcp [ 3852.292152] Key type .llcrypt unregistered [ 3852.295407] Key type ._llcrypt unregistered [ 3872.104951] Key type ._llcrypt registered [ 3872.108863] Key type .llcrypt registered [ 3872.293779] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3873.447377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3873.628987] alg: No test for adler32 (adler32-zlib) [ 3874.691945] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 3874.888520] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3876.544992] Key type lgssc registered [ 3877.718501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3885.198815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3885.234696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3886.760336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.312855] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3896.298123] LustreError: 120992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3896.331774] LustreError: 120992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3899.907888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.236928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3905.117182] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3913.293715] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3919.343406] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3926.332200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3926.658173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.808781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3932.817988] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3940.391491] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3949.987953] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3953.495363] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3953.504543] Lustre: Skipped 2 previous similar messages [ 3956.880947] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3956.954976] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3958.323831] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3958.328770] Lustre: Skipped 2 previous similar messages [ 3965.032525] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3965.037028] Lustre: Skipped 1 previous similar message [ 3970.070570] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.33@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3970.079544] Lustre: Skipped 2 previous similar messages [ 3975.217036] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.33@tcp 192.168.0.[2,10]@tcp [ 3983.136149] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.33/24@tcp in log lustre-MDT0000 [ 3983.143658] Lustre: Skipped 2 previous similar messages [ 3983.569453] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.33/24@tcp [ 3983.575398] Lustre: Skipped 1 previous similar message [ 3991.538553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3991.540398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.545524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3991.557661] Lustre: Skipped 1 previous similar message [ 3992.514792] Lustre: server umount lustre-OST0000 complete [ 3996.643364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.647310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3996.661763] Lustre: Skipped 1 previous similar message [ 3996.690274] Lustre: Skipped 2 previous similar messages [ 4001.761134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.775574] Lustre: Skipped 1 previous similar message [ 4002.417087] Lustre: server umount lustre-MDT0000 complete [ 4006.660980] LustreError: 120974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753366 with bad export cookie 177179857965284296 [ 4006.668500] LustreError: 120974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4006.668586] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4006.939462] Lustre: server umount lustre-MDT0001 complete [ 4016.840713] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4019.496909] Key type lgssc unregistered [ 4019.747694] LNet: 125797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4019.753869] LNetError: 125797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4019.769571] LNet: Removed LNI 192.168.201.133@tcp [ 4020.474217] Key type .llcrypt unregistered [ 4020.479527] Key type ._llcrypt unregistered [ 4045.340975] Key type ._llcrypt registered [ 4045.347271] Key type .llcrypt registered [ 4045.456965] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4046.603627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4046.626644] alg: No test for adler32 (adler32-zlib) [ 4047.711405] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 4048.048789] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4049.839173] Key type lgssc registered [ 4051.067342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4058.656560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4058.692094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.059217] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4060.084212] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.33/24@tcp [ 4060.091502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.400886] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4070.374125] LustreError: 127353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.384706] LustreError: 127353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4072.099270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4072.368584] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4072.388524] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.33/24@tcp [ 4072.440311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4076.696838] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4083.471292] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4088.786256] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4095.381937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4095.730144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.096289] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4105.041587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4110.714837] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4123.622827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4123.624148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.625817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4123.668204] Lustre: Skipped 1 previous similar message [ 4127.887599] Lustre: server umount lustre-OST0000 complete [ 4133.856540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4133.859914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.866688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4133.866695] Lustre: Skipped 1 previous similar message [ 4133.893638] Lustre: Skipped 1 previous similar message [ 4138.976607] LustreError: 127353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4139.003582] LustreError: 127353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4140.110045] Lustre: server umount lustre-MDT0000 complete [ 4143.094517] LustreError: 128403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.105957] LustreError: 128403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4143.867162] LustreError: 127334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753504 with bad export cookie 11970390511642286084 [ 4143.877800] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4144.192208] Lustre: server umount lustre-MDT0001 complete [ 4156.186923] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4159.190802] Key type lgssc unregistered [ 4159.523532] LNet: 130948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4159.536892] LNetError: 130948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4159.559532] LNet: Removed LNI 192.168.201.133@tcp [ 4160.480494] Key type .llcrypt unregistered [ 4160.483594] Key type ._llcrypt unregistered [ 4187.591885] Key type ._llcrypt registered [ 4187.593925] Key type .llcrypt registered [ 4187.695264] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 4205.529348] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4207.072817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4207.166168] alg: No test for adler32 (adler32-zlib) [ 4208.182557] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 4208.433561] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4210.096209] Key type lgssc registered [ 4211.275681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4249.065995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4249.100439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.243170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4250.259537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4250.324982] Lustre: lustre-MDT0000: new disk, initializing [ 4250.421533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4250.436120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4256.031232] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4267.326948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4267.567870] Lustre: 134351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4267.651638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4267.655638] Lustre: Skipped 1 previous similar message [ 4267.767681] Lustre: lustre-MDT0001: new disk, initializing [ 4267.917960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4267.992697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4268.024679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4273.402950] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4283.654898] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4289.508519] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4297.985473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4298.275913] Lustre: lustre-OST0000: new disk, initializing [ 4298.280926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4298.290362] Lustre: 136323:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4298.401522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.436418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4304.446508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4304.574835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4305.032709] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4314.974985] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4319.916805] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4320.196981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4325.912877] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4326.285866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4330.466357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4330.479729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.483397] LustreError: Skipped 1 previous similar message [ 4330.488288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4330.506541] Lustre: Skipped 1 previous similar message [ 4330.526177] Lustre: Skipped 1 previous similar message [ 4335.748920] Lustre: server umount lustre-OST0000 complete [ 4340.704947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4340.710893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4340.727286] Lustre: Skipped 1 previous similar message [ 4345.825571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4345.827829] Lustre: Skipped 2 previous similar messages [ 4347.159415] Lustre: server umount lustre-MDT0000 complete [ 4350.731394] LustreError: 134343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753710 with bad export cookie 10797533589498337187 [ 4350.739224] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4350.742602] LustreError: 134343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4351.055911] Lustre: server umount lustre-MDT0001 complete [ 4360.738734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.107747] LustreError: 138541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.133617] LustreError: 138541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4361.223908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4366.305965] LustreError: 138542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.602789] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4371.431182] LustreError: 138541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4376.546878] LustreError: 138542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.538430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.982732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4384.785825] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4393.198030] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4402.823417] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4412.293630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4412.652605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.280632] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4427.592872] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4433.587601] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4433.741881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4439.296564] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4439.523110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4449.624506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4449.960615] Lustre: lustre-OST0001: new disk, initializing [ 4449.969243] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4449.979903] Lustre: 141919:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4450.090427] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4456.074506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4456.080883] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4456.186582] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4456.882364] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4466.456160] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4466.703793] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4471.820274] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4472.089519] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4478.889801] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d5444719000.ost_server_uuid 50 [ 4480.467684] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d5444719000.ost_server_uuid in IDLE state after 0 sec [ 4492.257205] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4492.263885] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4492.277832] LustreError: 140368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.288546] LustreError: 140368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4492.336624] Lustre: server umount lustre-OST0001 complete [ 4495.841191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4495.852819] LustreError: Skipped 1 previous similar message [ 4495.859839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4495.868522] Lustre: Skipped 1 previous similar message [ 4495.875773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4502.500487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4502.501597] LustreError: 140366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.513423] Lustre: Skipped 3 previous similar messages [ 4502.531197] LustreError: 140366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4510.175462] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4510.369616] Lustre: server umount lustre-OST0000 complete [ 4517.344338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4517.362712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.379405] Lustre: Skipped 1 previous similar message [ 4517.386502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4517.393095] Lustre: Skipped 2 previous similar messages [ 4520.810399] Lustre: server umount lustre-MDT0000 complete [ 4522.978909] LustreError: 138536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.003770] LustreError: 138536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4524.518181] LustreError: 138521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753884 with bad export cookie 10797533589498338111 [ 4524.521391] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4524.530682] LustreError: 138521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4524.986217] Lustre: server umount lustre-MDT0001 complete [ 4546.603776] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4549.695680] Key type lgssc unregistered [ 4549.974973] LNet: 144926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4549.987507] LNetError: 144926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4550.005496] LNet: Removed LNI 192.168.201.133@tcp [ 4550.982420] Key type .llcrypt unregistered [ 4550.988346] Key type ._llcrypt unregistered [ 4596.239724] Key type ._llcrypt registered [ 4596.241119] Key type .llcrypt registered [ 4596.353898] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 4614.282176] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4615.860576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4615.899178] alg: No test for adler32 (adler32-zlib) [ 4616.996317] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 4617.272635] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4618.976838] Key type lgssc registered [ 4620.473668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4658.767810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4658.808203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4660.009297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4660.041441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4660.183358] Lustre: lustre-MDT0000: new disk, initializing [ 4660.343966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4660.378534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4664.916515] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4676.682429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4676.867142] Lustre: 149081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4676.944238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4676.953139] Lustre: Skipped 1 previous similar message [ 4677.073882] Lustre: lustre-MDT0001: new disk, initializing [ 4677.232170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4677.312511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4677.342525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4682.964444] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4694.955313] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4700.650747] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4708.143755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4708.496118] Lustre: lustre-OST0000: new disk, initializing [ 4708.505430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4708.531824] Lustre: 151055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4708.667805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.614731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4715.630344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4715.674381] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4715.817859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4726.470143] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4730.751485] Lustre: Failing over lustre-OST0000 [ 4730.886942] Lustre: server umount lustre-OST0000 complete [ 4747.744543] Lustre: 147306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786754091/real 1786754091] req@00000000a8b168ae x1873547338224640/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786754107 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4747.745195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4747.756337] Lustre: 147306:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4747.786953] Lustre: Skipped 1 previous similar message [ 4748.767338] Lustre: 147304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786754093/real 1786754093] req@0000000025c9ad49 x1873547338225408/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786754109 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4750.334442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.717076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4750.747144] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4752.736302] Lustre: 147305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786754096/real 1786754096] req@00000000a8512214 x1873547338226560/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786754112 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4752.805119] Lustre: 147305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4756.835023] Lustre: 147306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786754101/real 1786754101] req@0000000058c259fd x1873547338227712/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786754117 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4757.139893] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4757.877357] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4759.262160] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4759.262979] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4761.144140] Lustre: Failing over lustre-MDT0000 [ 4761.394674] Lustre: server umount lustre-MDT0000 complete [ 4762.079505] Lustre: 147305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786754106/real 1786754106] req@0000000015e92947 x1873547338228736/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786754122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4762.096609] Lustre: 147305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4763.105520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4763.120682] Lustre: Skipped 1 previous similar message [ 4768.233866] LustreError: 150273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4768.261908] LustreError: 150273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4769.452408] LustreError: 149090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.346760] LustreError: 149088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.369659] LustreError: 149088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4778.468805] LustreError: 149093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4778.493484] LustreError: 149093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4779.488642] Lustre: 147306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786754123/real 1786754123] req@0000000015e92947 x1873547338238592/t0(0) o400->MGC192.168.201.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786754139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4779.522291] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4780.898343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.610296] LustreError: 149093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.649582] LustreError: 149093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4789.728971] LustreError: 147302:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000ad34eef x1873547338244096/t0(0) o250->MGC192.168.201.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4790.193451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.288604] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4790.954577] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4795.237823] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4795.369605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4795.377108] Lustre: Skipped 1 previous similar message [ 4795.390289] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4795.443835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4805.616634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4805.621279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4805.623563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4805.631041] Lustre: Skipped 2 previous similar messages [ 4810.726119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4810.748822] Lustre: Skipped 1 previous similar message [ 4811.205365] Lustre: server umount lustre-OST0000 complete [ 4820.970655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.985212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4820.992808] Lustre: Skipped 1 previous similar message [ 4820.995323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4820.997634] Lustre: Skipped 1 previous similar message [ 4822.791599] Lustre: server umount lustre-MDT0000 complete [ 4826.082133] LustreError: 150273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4826.110649] LustreError: 150273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4826.963736] LustreError: 149074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754187 with bad export cookie 882814113913696183 [ 4826.968603] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4826.975629] LustreError: 149074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4827.279133] Lustre: server umount lustre-MDT0001 complete [ 4841.812816] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4845.362678] Key type lgssc unregistered [ 4845.774131] LNet: 154714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4845.785329] LNetError: 154714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4845.809295] LNet: Removed LNI 192.168.201.133@tcp [ 4846.997725] Key type .llcrypt unregistered [ 4846.999904] Key type ._llcrypt unregistered [ 4886.778291] Key type ._llcrypt registered [ 4886.780921] Key type .llcrypt registered [ 4886.930559] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4888.199940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4888.246716] alg: No test for adler32 (adler32-zlib) [ 4889.323886] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 4889.632193] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4891.351958] Key type lgssc registered [ 4892.894578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4900.576237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4900.595569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.215597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.329254] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4912.609927] LustreError: 156235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4912.646661] LustreError: 156235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4917.729531] LustreError: 156236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4918.262573] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4922.387762] LustreError: 156264:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4922.398207] LustreError: 156264:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4922.405950] LustreError: 156264:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4923.100992] Lustre: server umount lustre-MDT0000 complete [ 4942.348652] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 4954.570945] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4994.269627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4994.518895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4994.548686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4994.635824] Lustre: lustre-MDT0000: new disk, initializing [ 4994.751449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4994.770630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5000.079452] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5011.611810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.723563] Lustre: 159569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5011.771573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5011.775450] Lustre: Skipped 1 previous similar message [ 5011.843476] Lustre: lustre-MDT0001: new disk, initializing [ 5011.942731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5011.977062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5011.986924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5016.229383] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5026.938461] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5032.507958] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5039.973332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5040.156743] Lustre: lustre-OST0000: new disk, initializing [ 5040.161623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5040.167234] Lustre: 161543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5040.238282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5046.274613] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5049.960536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5049.976975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5050.082067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5056.240835] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5060.244234] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5062.067752] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:40:20 (1786754420) [ 5321.697075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.699649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5321.710924] Lustre: Skipped 2 previous similar messages [ 5321.724541] Lustre: Skipped 1 previous similar message [ 5326.816580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5326.833983] Lustre: Skipped 3 previous similar messages [ 5327.792703] Lustre: server umount lustre-MDT0000 complete [ 5331.942309] LustreError: 159579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.982033] LustreError: 159579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5333.871241] LustreError: 159562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754694 with bad export cookie 15271784916321798660 [ 5333.884934] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.889065] LustreError: 159562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5337.055587] LustreError: 159578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5337.058649] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.080787] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5343.199937] LustreError: 163305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5343.205589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5343.212276] LustreError: 163305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5343.232528] Lustre: Skipped 1 previous similar message [ 5348.323067] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5348.893682] Lustre: server umount lustre-MDT0001 complete [ 5359.419377] Lustre: server umount lustre-OST0000 complete [ 5368.268725] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 5378.060360] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5416.540223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.722250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5416.741916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5416.824418] Lustre: lustre-MDT0000: new disk, initializing [ 5416.922528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5416.937397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5423.033540] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5438.589611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5438.807481] Lustre: 166210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5438.888431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5438.892786] Lustre: Skipped 1 previous similar message [ 5439.009481] Lustre: lustre-MDT0001: new disk, initializing [ 5439.228240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5439.291598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5439.311804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5444.973853] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5455.323468] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5462.111581] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5469.117433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5469.336557] Lustre: lustre-OST0000: new disk, initializing [ 5469.339524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5469.344323] Lustre: 168181:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5469.421099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5476.109863] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5477.474504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5477.484360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5477.640843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5485.575640] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5490.395139] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5490.606268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5496.156630] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5496.432807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5498.340147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5498.341093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5498.347179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5498.347190] Lustre: Skipped 1 previous similar message [ 5498.354834] LustreError: Skipped 1 previous similar message [ 5498.365900] Lustre: Skipped 1 previous similar message [ 5504.404735] Lustre: server umount lustre-OST0000 complete [ 5511.141182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5511.149708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5511.164745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5511.167625] Lustre: Skipped 3 previous similar messages [ 5515.393248] Lustre: server umount lustre-MDT0000 complete [ 5518.524080] LustreError: 166202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754878 with bad export cookie 15271784916321957532 [ 5518.532792] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5518.793488] Lustre: server umount lustre-MDT0001 complete [ 5527.864616] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:48:06 (1786754886) [ 5546.724267] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 5558.459571] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5598.600676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.995431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5599.098115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5599.323526] Lustre: lustre-MDT0000: new disk, initializing [ 5599.407504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.433595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5603.964674] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5621.011708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5621.286480] Lustre: 172939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5621.826884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5627.616556] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5639.626868] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5646.360253] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5653.530222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5653.841466] Lustre: lustre-OST0000: new disk, initializing [ 5653.844615] Lustre: Skipped 1 previous similar message [ 5653.851314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5653.864888] Lustre: Skipped 2 previous similar messages [ 5653.894082] Lustre: 174919:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5660.730062] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5661.242186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5661.248550] Lustre: Skipped 1 previous similar message [ 5661.252232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5661.430136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5671.910594] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5682.146430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5682.148769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.162695] LustreError: Skipped 1 previous similar message [ 5682.165722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5682.180316] Lustre: Skipped 2 previous similar messages [ 5682.206535] Lustre: Skipped 3 previous similar messages [ 5684.561574] Lustre: server umount lustre-OST0000 complete [ 5692.386804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5694.658250] Lustre: server umount lustre-MDT0000 complete [ 5697.512050] LustreError: 174056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5697.542865] LustreError: 174056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5698.247418] LustreError: 172931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755058 with bad export cookie 15271784916321958456 [ 5698.266583] LustreError: 172931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5698.288493] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5698.577850] Lustre: server umount lustre-MDT0001 complete [ 5709.106135] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:51:07 (1786755067) [ 5729.717433] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 5739.662872] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5778.196197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.517296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5778.548022] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5778.625726] Lustre: lustre-MDT0000: new disk, initializing [ 5778.732916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.735924] Lustre: Skipped 2 previous similar messages [ 5778.777735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5782.498329] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5793.075844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.233568] Lustre: 179250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5793.594816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5797.943890] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5807.950817] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5813.497962] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5819.539232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5819.924754] Lustre: 181229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5825.655432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5825.888601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5826.942656] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5837.685357] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5846.498866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5846.504978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5846.508256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5846.508265] Lustre: Skipped 2 previous similar messages [ 5846.532574] Lustre: Skipped 2 previous similar messages [ 5852.461329] Lustre: server umount lustre-OST0000 complete [ 5862.947699] Lustre: server umount lustre-MDT0000 complete [ 5866.679047] LustreError: 179243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755226 with bad export cookie 15271784916321959583 [ 5866.687791] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5866.692964] LustreError: 179243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5867.055698] Lustre: server umount lustre-MDT0001 complete [ 5880.254217] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5883.130702] Key type lgssc unregistered [ 5883.371935] LNet: 183089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5883.380163] LNetError: 183089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5883.430568] LNet: Removed LNI 192.168.201.133@tcp [ 5884.189092] Key type .llcrypt unregistered [ 5884.191107] Key type ._llcrypt unregistered [ 5909.115229] Key type ._llcrypt registered [ 5909.117298] Key type .llcrypt registered [ 5909.225570] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5910.337422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5910.370101] alg: No test for adler32 (adler32-zlib) [ 5911.491408] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 5911.730748] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 5913.367178] Key type lgssc registered [ 5914.253421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5922.397097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5922.414056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5923.865342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5928.670666] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5934.049598] LustreError: 184504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.054366] LustreError: 184504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5937.494384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5937.840384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5942.204461] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5950.613637] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5956.125350] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5963.718845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5964.080284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.218563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5971.626093] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5981.108490] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6004.198625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6004.198799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.201467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6004.221450] Lustre: Skipped 1 previous similar message [ 6007.173317] Lustre: server umount lustre-OST0000 complete [ 6014.432103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6014.435035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6014.440501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6014.458823] Lustre: Skipped 2 previous similar messages [ 6014.473232] Lustre: Skipped 1 previous similar message [ 6019.553036] LustreError: 184505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6019.565330] LustreError: 184505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6019.653511] Lustre: server umount lustre-MDT0000 complete [ 6023.413601] LustreError: 184486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755383 with bad export cookie 12096336526344601556 [ 6023.418151] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6023.421897] LustreError: 184486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6023.858862] Lustre: server umount lustre-MDT0001 complete [ 6037.137513] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 6040.186176] Key type lgssc unregistered [ 6040.579227] LNet: 188068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6040.587395] LNetError: 188068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6040.606080] LNet: Removed LNI 192.168.201.133@tcp [ 6041.568820] Key type .llcrypt unregistered [ 6041.571148] Key type ._llcrypt unregistered [ 6066.540210] Key type ._llcrypt registered [ 6066.543171] Key type .llcrypt registered [ 6066.632825] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6067.535987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6067.625922] alg: No test for adler32 (adler32-zlib) [ 6068.679279] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 6068.917922] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 6070.599624] Key type lgssc registered [ 6071.678729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6079.702883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6079.743487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6081.125380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6086.494433] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6091.752169] LustreError: 189486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6091.779441] LustreError: 189486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6096.076354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6096.560993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6101.525692] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6110.245358] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6116.620398] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6126.917789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6127.496566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6128.534394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6135.506447] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6143.554848] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6150.357171] Lustre: lustre-OST0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 6151.142439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6151.142862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.147875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6151.186031] Lustre: Skipped 1 previous similar message [ 6155.443432] Lustre: lustre-OST0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 6155.450669] Lustre: Skipped 1 previous similar message [ 6156.117700] Lustre: server umount lustre-OST0000 complete [ 6166.058794] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d5447b9b800.ost_server_uuid 50 [ 6168.402204] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6183.393754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6183.418256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.453942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6185.832882] Lustre: server umount lustre-MDT0000 complete [ 6186.978096] LustreError: 189486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6186.978214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.676334] LustreError: 189468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755550 with bad export cookie 5396755841178982016 [ 6190.677105] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6190.689551] LustreError: 189468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6190.997790] Lustre: server umount lustre-MDT0001 complete [ 6201.118598] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:59:19 (1786755559) [ 6209.544317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.191552] LustreError: 193406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6210.215464] LustreError: 193406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6210.290767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.609854] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6215.659624] LustreError: 193407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6219.748560] LustreError: 193406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6224.938204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6225.348534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6230.581762] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6240.119863] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6246.487755] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6252.972523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6253.317315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.418674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6259.060445] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6267.980409] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6275.599241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6275.816634] Lustre: lustre-OST0001: new disk, initializing [ 6275.818392] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6275.827609] Lustre: 196250:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6275.911568] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6283.525993] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6284.892447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6284.915426] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6285.081440] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6295.699585] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6300.650934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.669449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.673359] Lustre: Skipped 1 previous similar message [ 6300.696606] Lustre: Skipped 1 previous similar message [ 6304.920657] Lustre: server umount lustre-OST0000 complete [ 6305.762648] LustreError: 196833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.805946] LustreError: 196833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6312.827753] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6313.057361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6318.566781] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6318.807804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6321.829339] LustreError: 196945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6321.845204] LustreError: 196945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6336.486158] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6336.492816] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.500613] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6336.503652] Lustre: Skipped 2 previous similar messages [ 6339.806524] Lustre: server umount lustre-OST0001 complete [ 6346.732162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6346.749317] Lustre: Skipped 2 previous similar messages [ 6350.646420] Lustre: server umount lustre-MDT0000 complete [ 6355.434689] LustreError: 193387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755715 with bad export cookie 5396755841178983087 [ 6355.445121] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6355.458086] LustreError: 193387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6355.783924] Lustre: server umount lustre-MDT0001 complete [ 6385.365629] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:02:23 (1786755743) [ 6392.180518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.521745] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6392.571576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6392.782773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6398.122133] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6407.223846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6407.372728] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6407.451789] Lustre: 199705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6413.122428] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6420.599732] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6426.377845] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6432.383811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6432.508944] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6438.563180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6439.069582] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6447.087713] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6454.945505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6455.044937] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6461.747471] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6469.157875] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6474.211886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6474.215702] LustreError: Skipped 1 previous similar message [ 6474.234103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.262254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6474.268982] Lustre: Skipped 2 previous similar messages [ 6478.236473] Lustre: server umount lustre-OST0000 complete [ 6480.058492] LustreError: 201557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6480.070058] LustreError: 201557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6497.263458] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6497.274694] LustreError: Skipped 1 previous similar message [ 6500.287187] Lustre: server umount lustre-OST0001 complete [ 6507.487971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6507.491839] Lustre: Skipped 7 previous similar messages [ 6507.494896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6507.506598] Lustre: Skipped 4 previous similar messages [ 6510.682169] Lustre: server umount lustre-MDT0000 complete [ 6514.530675] LustreError: 200452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755874 with bad export cookie 5396755841178984494 [ 6514.531472] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6514.540381] LustreError: 200452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6514.803355] Lustre: server umount lustre-MDT0001 complete [ 6538.608239] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:04:57 (1786755897) [ 6554.729921] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 6566.136749] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6604.538385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6604.808812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6604.867591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6604.953271] Lustre: lustre-MDT0000: new disk, initializing [ 6605.050903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6605.061615] Lustre: Skipped 3 previous similar messages [ 6605.074691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6609.821502] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6620.584775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.816417] Lustre: 207800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6620.918527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6620.932316] Lustre: Skipped 1 previous similar message [ 6621.054636] Lustre: lustre-MDT0001: new disk, initializing [ 6621.195535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6621.215121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6625.384630] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6634.052161] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6639.871894] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6647.196820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.469152] Lustre: lustre-OST0000: new disk, initializing [ 6647.474547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6647.479647] Lustre: 209771:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6653.963528] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6654.560218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6654.566346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6654.704913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6663.287492] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6669.537097] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6669.802457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6674.203890] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6674.374065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6680.547107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.551146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6680.575158] Lustre: Skipped 1 previous similar message [ 6680.591764] Lustre: Skipped 2 previous similar messages [ 6682.767277] Lustre: server umount lustre-OST0000 complete [ 6687.712718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6687.719844] LustreError: Skipped 2 previous similar messages [ 6692.906770] Lustre: server umount lustre-MDT0000 complete [ 6695.907133] LustreError: 207808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.936752] LustreError: 207808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6697.299473] LustreError: 207792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756057 with bad export cookie 5396755841178986048 [ 6697.312860] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6697.314508] LustreError: 207792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6697.866561] Lustre: server umount lustre-MDT0001 complete [ 6706.235490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.357977] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6724.844611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6730.111731] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6739.212540] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6745.589655] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6753.221322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6753.625053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6753.634299] Lustre: Skipped 4 previous similar messages [ 6761.186457] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6769.755801] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6774.845920] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6778.496689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6785.175547] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6785.468886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6790.637565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6793.792172] Lustre: server umount lustre-OST0000 complete [ 6801.254723] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6806.103974] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6837.578196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6843.268558] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6843.281744] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6845.343744] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6853.782822] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6855.514566] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6858.622823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6858.665768] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6863.845513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.851405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6863.856444] Lustre: Skipped 5 previous similar messages [ 6863.867425] Lustre: Skipped 7 previous similar messages [ 6865.880234] Lustre: server umount lustre-OST0000 complete [ 6873.573306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6873.586283] LustreError: Skipped 1 previous similar message [ 6884.320381] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6884.563718] Lustre: server umount lustre-MDT0000 complete [ 6888.447514] LustreError: 211966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756248 with bad export cookie 5396755841178986979 [ 6888.448071] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6888.461069] LustreError: 211966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6888.819406] Lustre: server umount lustre-MDT0001 complete [ 6898.203786] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:10:56 (1786756256) [ 6904.599555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6904.904095] LustreError: 217983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6904.911559] LustreError: 217983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6909.402269] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6917.433521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.002971] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6928.250771] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6934.352706] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6940.679973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.140214] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6954.564086] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6959.244303] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6959.412675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6963.783640] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6964.103771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6970.855572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.973664] Lustre: lustre-OST0001: new disk, initializing [ 6970.977724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6970.982189] Lustre: 221355:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6977.038363] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6977.091846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6977.101244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6977.209809] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6986.828927] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6991.433118] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6991.667921] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6995.939857] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6996.193553] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7003.106704] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7004.452494] Lustre: server umount lustre-OST0001 complete [ 7012.209461] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7012.581241] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7018.095949] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7018.400131] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7048.097759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7048.258568] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7048.265368] Lustre: Skipped 5 previous similar messages [ 7053.289067] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7053.302026] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7053.302725] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7053.334449] Lustre: Skipped 1 previous similar message [ 7054.932682] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7062.217449] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7063.832616] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7072.280297] Lustre: server umount lustre-OST0001 complete [ 7095.678197] Lustre: server umount lustre-OST0000 complete [ 7105.601119] Lustre: server umount lustre-MDT0000 complete [ 7110.530716] LustreError: 221354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756470 with bad export cookie 5396755841178987938 [ 7110.542072] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7110.543915] LustreError: 221354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7117.066974] Lustre: server umount lustre-MDT0001 complete [ 7146.991522] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:15:05 (1786756505) [ 7154.405751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7154.594294] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7154.631676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7159.153043] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7168.127114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7168.253429] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7168.295581] Lustre: 226599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7173.395451] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7181.080972] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7187.692499] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7194.125245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7194.258135] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7201.793630] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7210.098696] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7220.469693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7220.582825] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7227.210938] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7235.067546] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7236.557334] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7241.613319] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7241.803639] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7246.066774] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7246.295303] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7252.629636] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d54459ae800.ost_server_uuid 50 [ 7254.403165] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d54459ae800.ost_server_uuid in IDLE state after 0 sec [ 7255.390913] Lustre: Permanently deactivating lustre-OST0001 [ 7255.414106] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7258.553896] Lustre: Permanently reactivating lustre-OST0001 [ 7264.743804] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.769515] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7264.784178] Lustre: Skipped 10 previous similar messages [ 7264.789317] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7264.798522] LustreError: Skipped 5 previous similar messages [ 7264.815615] Lustre: Skipped 20 previous similar messages [ 7265.520309] Lustre: server umount lustre-OST0001 complete [ 7269.859558] LustreError: 228448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7269.883512] LustreError: 228448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 7276.751506] Lustre: server umount lustre-OST0000 complete [ 7288.055891] Lustre: server umount lustre-MDT0000 complete [ 7292.658293] LustreError: 226590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756652 with bad export cookie 5396755841178989233 [ 7292.663695] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7292.673390] LustreError: 226590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7293.469727] Lustre: server umount lustre-MDT0001 complete [ 7322.254802] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:18:00 (1786756680) [ 7331.539744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.955196] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7331.980378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7331.989396] Lustre: Skipped 5 previous similar messages [ 7338.392146] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7347.207523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.311842] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7347.348362] Lustre: 233293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7353.233322] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7361.936034] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7368.831576] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7377.170127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.427491] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7377.455358] Lustre: Permanently deactivating lustre-OST0000 [ 7377.478234] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7384.732443] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7386.211354] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7394.093872] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7401.972768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7402.107137] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7409.279938] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7411.471539] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7419.821778] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7422.356783] Lustre: Permanently reactivating lustre-OST0000 [ 7422.395283] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7422.398581] Lustre: Skipped 2 previous similar messages [ 7430.843185] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 45 seconds. I think it's dead, and I am evicting it. exp 00000000682b5c0f, cur 1786756791 deadline 1786756776 last 1786756746 [ 7433.053878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7442.332976] Lustre: server umount lustre-OST0001 complete [ 7453.704779] Lustre: server umount lustre-OST0000 complete [ 7465.751953] Lustre: server umount lustre-MDT0000 complete [ 7469.640460] LustreError: 233287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756829 with bad export cookie 5396755841178990913 [ 7469.659238] LustreError: 233287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7469.659649] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7469.988988] Lustre: server umount lustre-MDT0001 complete [ 7482.590311] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 7486.776561] Key type lgssc unregistered [ 7487.158604] LNet: 238171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7487.182701] LNetError: 238171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7488.230376] LNet: Removed LNI 192.168.201.133@tcp [ 7489.579219] Key type .llcrypt unregistered [ 7489.582935] Key type ._llcrypt unregistered [ 7516.838446] Key type ._llcrypt registered [ 7516.848609] Key type .llcrypt registered [ 7516.965764] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7518.384187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7518.518452] alg: No test for adler32 (adler32-zlib) [ 7519.663950] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 7520.036377] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 7521.847211] Key type lgssc registered [ 7523.253389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7535.070150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7535.089546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.636815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.145645] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7545.828207] LustreError: 239643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7545.852263] LustreError: 239643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7550.959557] LustreError: 239642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7553.297286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7553.494541] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7553.544923] Lustre: 239630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7553.581563] Lustre: Permanently deactivating lustre-MDT0001 [ 7553.621879] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7554.125355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7559.581824] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7561.792163] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7571.530784] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7579.925980] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7587.193732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7587.503219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7594.749150] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7598.763704] LustreError: 241488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7604.208576] LustreError: 241974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7604.944632] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7606.289124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7611.365134] LustreError: 241486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7611.373234] LustreError: 241486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7612.716846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7613.097776] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7618.555510] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7618.575824] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7619.069846] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7627.297840] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7630.578553] Lustre: Permanently reactivating lustre-MDT0001 [ 7630.594483] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7630.598738] Lustre: Skipped 2 previous similar messages [ 7633.361649] Lustre: lustre-MDT0001: Client 1f51acbd-4c98-4ab0-8e9d-b4f40938c307 (at 192.168.201.33@tcp) reconnecting [ 7636.071481] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7636.083487] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7636.085923] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7636.100319] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7636.102366] LustreError: 243275:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7636.114868] LustreError: 243275:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7636.120404] LustreError: 243275:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.133@tcp: cfg command failed: rc = -108 [ 7636.123460] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7636.123460] [ 7636.129263] LustreError: 239634:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7639.608860] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7641.637827] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7648.607383] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7648.871543] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7654.256270] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7654.520959] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7657.428369] Lustre: Permanently deactivating lustre-MDT0001 [ 7657.455468] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7657.464290] Lustre: Skipped 2 previous similar messages [ 7663.165994] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7672.801824] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7672.814883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7672.826380] Lustre: Skipped 3 previous similar messages [ 7672.848511] Lustre: Skipped 3 previous similar messages [ 7677.924232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7677.938622] Lustre: Skipped 3 previous similar messages [ 7678.347483] Lustre: server umount lustre-MDT0000 complete [ 7682.243993] LustreError: 239624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757042 with bad export cookie 13362741450850391107 [ 7682.248682] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7682.260150] LustreError: 239624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7682.712585] Lustre: server umount lustre-MDT0001 complete [ 7689.022106] Lustre: server umount lustre-OST0000 complete [ 7701.937757] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:24:20 (1786757060) [ 7713.951969] Lustre: server umount lustre-OST0001 complete [ 7721.786878] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 7733.167653] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7778.943590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7779.208892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7779.225728] Lustre: Skipped 2 previous similar messages [ 7779.266638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7779.365637] Lustre: lustre-MDT0000: new disk, initializing [ 7779.493669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.518927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7783.680815] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7795.658386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7795.784916] Lustre: 247907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7795.846228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7795.848631] Lustre: Skipped 1 previous similar message [ 7796.019981] Lustre: lustre-MDT0001: new disk, initializing [ 7796.224894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7796.323039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7796.353730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7802.238149] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7815.765894] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7823.447376] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7830.142942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7830.638682] Lustre: lustre-OST0000: new disk, initializing [ 7830.648238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7830.666092] Lustre: 249881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7830.814267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7837.022747] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7840.414871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7840.430363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7840.604858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7847.845612] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7850.260110] LustreError: 249135:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7860.287209] LustreError: 249135:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7860.336555] LustreError: 250650:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7860.347745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7860.628500] Lustre: lustre-OST0001: new disk, initializing [ 7860.636680] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7860.649923] Lustre: 250997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7866.464360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7866.474408] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7866.583500] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7867.264462] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7870.383204] LustreError: 250650:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7877.431571] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7879.503670] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7882.208725] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7882.211089] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7882.215515] LustreError: Skipped 1 previous similar message [ 7882.221617] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7882.231563] Lustre: Skipped 1 previous similar message [ 7882.253944] Lustre: Skipped 1 previous similar message [ 7887.336138] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7887.352668] Lustre: Skipped 2 previous similar messages [ 7888.188143] Lustre: server umount lustre-OST0001 complete [ 7892.455728] LustreError: 249864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7892.466812] LustreError: 249864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7907.297574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7907.307462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7907.314694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7907.324167] Lustre: Skipped 1 previous similar message [ 7907.325140] Lustre: Skipped 1 previous similar message [ 7912.947778] Lustre: server umount lustre-OST0000 complete [ 7917.744645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.759432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7917.765603] Lustre: Skipped 1 previous similar message [ 7932.383832] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7932.749212] Lustre: server umount lustre-MDT0000 complete [ 7933.410134] LustreError: 247914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.431756] LustreError: 247914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7936.734600] LustreError: 247900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757296 with bad export cookie 13362741450850393704 [ 7936.738942] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7951.327244] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7952.038769] Lustre: server umount lustre-MDT0001 complete [ 7964.276204] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 7967.939725] Key type lgssc unregistered [ 7968.180293] LNet: 253039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7968.197842] LNetError: 253039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7968.229180] LNet: Removed LNI 192.168.201.133@tcp [ 7969.543164] Key type .llcrypt unregistered [ 7969.547746] Key type ._llcrypt unregistered [ 8016.239501] Key type ._llcrypt registered [ 8016.241420] Key type .llcrypt registered [ 8016.339967] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8017.393581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8017.426276] alg: No test for adler32 (adler32-zlib) [ 8018.417453] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 8018.563474] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8020.183161] Key type lgssc registered [ 8021.466944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8031.313796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8031.324489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8032.611106] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8032.628994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8032.805879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8038.958598] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8050.870054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8051.006417] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8051.045745] Lustre: 255247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8051.412986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8056.113607] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8064.521663] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8072.027242] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8079.937723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8080.254732] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8080.488332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8086.684951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8087.189708] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8097.329342] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8161.888815] Lustre: server umount lustre-OST0000 complete [ 8166.844826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8180.183141] Lustre: 254571:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786757524/real 1786757524] req@00000000c1a077fa x1873550904445440/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786757540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8180.209071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.467858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8182.821166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8184.806544] Lustre: 254573:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786757529/real 1786757529] req@0000000067960433 x1873550904446592/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786757545 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8184.836822] Lustre: 254573:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 8189.919995] Lustre: 254573:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786757534/real 1786757534] req@000000004708d9bf x1873550904447616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786757550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8189.955913] Lustre: 254573:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 8189.993125] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8190.012979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8190.022104] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8190.928856] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8198.876037] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8241.120313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8241.130123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8241.139072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8241.174507] Lustre: Skipped 2 previous similar messages [ 8246.242824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8246.248873] Lustre: Skipped 1 previous similar message [ 8246.444289] Lustre: server umount lustre-OST0000 complete [ 8251.362594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8251.367122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.374392] LustreError: Skipped 1 previous similar message [ 8251.398940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8251.398959] Lustre: Skipped 1 previous similar message [ 8256.480611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8256.493626] Lustre: Skipped 2 previous similar messages [ 8256.991646] Lustre: server umount lustre-MDT0000 complete [ 8261.166882] LustreError: 255242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757621 with bad export cookie 7516879267671017470 [ 8261.176230] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8261.179115] LustreError: 255242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8261.787386] Lustre: server umount lustre-MDT0001 complete [ 8274.588794] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8278.064505] Key type lgssc unregistered [ 8278.659503] LNet: 260683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8278.663026] LNetError: 260683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8278.703786] LNet: Removed LNI 192.168.201.133@tcp [ 8280.404955] Key type .llcrypt unregistered [ 8280.406513] Key type ._llcrypt unregistered [ 8309.745662] Key type ._llcrypt registered [ 8309.746702] Key type .llcrypt registered [ 8309.873451] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8311.590551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8311.745353] alg: No test for adler32 (adler32-zlib) [ 8313.078280] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 8313.579807] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8315.420459] Key type lgssc registered [ 8317.175416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8326.807080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8326.850462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8328.758671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8335.977385] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8337.914286] LustreError: 262102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8337.951247] LustreError: 262102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8343.016921] LustreError: 262103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8348.169769] LustreError: 262102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8349.194292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8349.688582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8355.814838] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8367.043132] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8374.445924] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8384.101876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8384.721896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8394.164587] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8399.123250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8403.171370] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8435.180314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8435.181174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8435.227536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8435.244325] Lustre: Skipped 1 previous similar message [ 8438.537678] Lustre: server umount lustre-OST0000 complete [ 8445.409918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8445.435697] Lustre: Skipped 1 previous similar message [ 8445.447504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8445.463595] Lustre: Skipped 1 previous similar message [ 8449.528731] Lustre: server umount lustre-MDT0000 complete [ 8450.529325] LustreError: 262103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.572632] LustreError: 262103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8454.465170] LustreError: 262081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757814 with bad export cookie 5581912156036536393 [ 8454.467141] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8454.476517] LustreError: 262081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8455.005932] Lustre: server umount lustre-MDT0001 complete [ 8467.678154] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8471.375814] Key type lgssc unregistered [ 8471.762847] LNet: 266418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8471.771977] LNetError: 266418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8471.785318] LNet: Removed LNI 192.168.201.133@tcp [ 8473.178261] Key type .llcrypt unregistered [ 8473.182328] Key type ._llcrypt unregistered [ 8494.357514] Key type ._llcrypt registered [ 8494.360926] Key type .llcrypt registered [ 8494.533889] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8496.009820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8496.062641] alg: No test for adler32 (adler32-zlib) [ 8497.188794] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 8497.506344] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8499.231442] Key type lgssc registered [ 8500.810788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8509.414438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8509.440682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8510.843693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.292937] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8521.186814] LustreError: 267642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.198955] LustreError: 267642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8524.378141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8525.018135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8530.859850] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8540.865198] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8549.566085] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8557.572750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8557.955862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8564.612492] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8576.065576] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8581.565372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8595.955709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8595.970392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8595.994843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8596.961267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8596.962230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8602.085403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8602.098641] Lustre: Skipped 1 previous similar message [ 8607.207173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8609.247189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8609.363526] Lustre: server umount lustre-OST0000 complete [ 8617.440993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8617.444304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8617.466630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8617.484153] Lustre: Skipped 1 previous similar message [ 8617.509394] Lustre: Skipped 2 previous similar messages [ 8620.974555] Lustre: server umount lustre-MDT0000 complete [ 8622.561498] LustreError: 268371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8622.604653] LustreError: 268371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8625.257244] LustreError: 270574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757985 with bad export cookie 7630484705283899194 [ 8625.267471] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8625.268581] LustreError: 270574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8625.766774] Lustre: server umount lustre-MDT0001 complete [ 8639.274687] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8642.122860] Key type lgssc unregistered [ 8642.582150] LNet: 271329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8642.588217] LNetError: 271329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8642.610586] LNet: Removed LNI 192.168.201.133@tcp [ 8643.568479] Key type .llcrypt unregistered [ 8643.570162] Key type ._llcrypt unregistered [ 8669.314799] Key type ._llcrypt registered [ 8669.317512] Key type .llcrypt registered [ 8669.451870] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8670.776711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8670.874387] alg: No test for adler32 (adler32-zlib) [ 8671.886885] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 8672.080501] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8673.736887] Key type lgssc registered [ 8674.852830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8683.392697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8683.410989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8684.904402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8690.270299] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8695.265868] LustreError: 272748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.281844] LustreError: 272748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8699.898495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.368759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8705.243092] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8713.233668] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8720.320529] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8727.957701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8728.414938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8730.478729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8736.173756] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8744.513822] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8778.819619] Lustre: server umount lustre-OST0000 complete [ 8785.906239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.912481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8785.937029] Lustre: Skipped 1 previous similar message [ 8785.976418] Lustre: Skipped 1 previous similar message [ 8791.009405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8791.017653] Lustre: Skipped 1 previous similar message [ 8794.233626] Lustre: server umount lustre-MDT0000 complete [ 8796.129154] LustreError: 272748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8796.154238] LustreError: 272748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8797.151285] Lustre: 272061:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758141/real 1786758141] req@00000000d82cb45c x1873551590198528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786758157 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8797.151365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8797.174654] Lustre: 272061:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8797.221888] LustreError: 272749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8797.264399] LustreError: 272749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8798.493992] LustreError: 272729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758158 with bad export cookie 8088004957672578713 [ 8798.509577] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8798.843200] Lustre: server umount lustre-MDT0001 complete [ 8810.943049] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8814.015138] Key type lgssc unregistered [ 8814.356705] LNet: 277060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8814.382488] LNetError: 277060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8814.408928] LNet: Removed LNI 192.168.201.133@tcp [ 8815.497363] Key type .llcrypt unregistered [ 8815.499630] Key type ._llcrypt unregistered [ 8833.806436] Key type ._llcrypt registered [ 8833.810149] Key type .llcrypt registered [ 8833.965985] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8835.663174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8835.709397] alg: No test for adler32 (adler32-zlib) [ 8836.907187] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 8837.177139] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8838.929286] Key type lgssc registered [ 8840.117419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8849.072568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8849.115777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8850.663327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8855.790746] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8860.144606] LustreError: 278297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8860.175661] LustreError: 278297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8863.821659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.228077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8868.839451] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8877.504800] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8884.346218] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8891.422829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8891.772284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8894.866228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8898.798455] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8907.788596] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8925.666278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8925.666941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8925.672237] LustreError: Skipped 1 previous similar message [ 8925.677487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8925.692163] Lustre: Skipped 1 previous similar message [ 8927.043389] Lustre: server umount lustre-OST0000 complete [ 8930.785787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8930.785931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8930.792300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8930.805312] Lustre: Skipped 1 previous similar message [ 8930.813721] Lustre: Skipped 2 previous similar messages [ 8935.906944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8935.913125] Lustre: Skipped 1 previous similar message [ 8936.730378] Lustre: server umount lustre-MDT0000 complete [ 8940.818991] LustreError: 278262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758301 with bad export cookie 7451085316982813583 [ 8940.825043] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8940.833614] LustreError: 278262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8941.236420] Lustre: server umount lustre-MDT0001 complete [ 8950.905936] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8953.547362] Key type lgssc unregistered [ 8953.991263] LNet: 282003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8953.999590] LNetError: 282003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8954.038137] LNet: Removed LNI 192.168.201.133@tcp [ 8954.921626] Key type .llcrypt unregistered [ 8954.923794] Key type ._llcrypt unregistered [ 8968.480245] Key type ._llcrypt registered [ 8968.482370] Key type .llcrypt registered [ 8968.553926] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8986.180389] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 9002.547733] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9003.766652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9003.990392] alg: No test for adler32 (adler32-zlib) [ 9005.231923] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 9005.557381] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 9007.231974] Key type lgssc registered [ 9008.557652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9049.918884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9050.009727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9051.249429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9051.277944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9051.384802] Lustre: lustre-MDT0000: new disk, initializing [ 9051.475120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9051.489770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9056.860992] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9069.267036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9069.401687] Lustre: 285562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9069.483806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9069.487795] Lustre: Skipped 1 previous similar message [ 9069.571725] Lustre: lustre-MDT0001: new disk, initializing [ 9069.641585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9069.682670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9069.687501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9075.458341] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9085.905234] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9091.944186] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9099.004782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9099.193726] Lustre: lustre-OST0000: new disk, initializing [ 9099.199990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9099.205749] Lustre: 287530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9099.280800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9105.077271] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9109.050725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9109.060837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9109.156619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9114.142720] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9118.722510] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9118.971712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9123.055711] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9123.297819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9129.954563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9129.956741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9129.966674] LustreError: Skipped 1 previous similar message [ 9129.973390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9130.002243] Lustre: Skipped 1 previous similar message [ 9131.586481] Lustre: server umount lustre-OST0000 complete [ 9136.098609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9136.110774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9136.123841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9136.128974] Lustre: Skipped 1 previous similar message [ 9140.193950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9140.198143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9140.226120] Lustre: Skipped 1 previous similar message [ 9141.554597] Lustre: server umount lustre-MDT0000 complete [ 9145.285336] LustreError: 285552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758505 with bad export cookie 8434598734848668718 [ 9145.309774] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9145.950331] Lustre: server umount lustre-MDT0001 complete [ 9153.878942] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:48:32 (1786758512) [ 9160.463971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9160.818655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9164.792934] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9166.307711] LustreError: 289943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.327747] LustreError: 289943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9171.441770] LustreError: 289944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9173.550453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9173.873398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9178.333895] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9185.074557] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9190.464356] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9197.851327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9198.173175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9204.388484] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9211.864846] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9222.117179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.118657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9222.136046] Lustre: Skipped 1 previous similar message [ 9222.145399] Lustre: Skipped 1 previous similar message [ 9226.851409] Lustre: server umount lustre-OST0000 complete [ 9230.303986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9230.310886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.320332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9236.473503] Lustre: server umount lustre-MDT0000 complete [ 9237.472836] LustreError: 289943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9237.485579] LustreError: 289943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9239.621330] LustreError: 289924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758599 with bad export cookie 8434598734848669649 [ 9239.628544] LustreError: 289924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9239.640581] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9239.802778] Lustre: server umount lustre-MDT0001 complete [ 9249.118620] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 9251.455325] Key type lgssc unregistered [ 9251.725934] LNet: 293482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9251.730450] LNetError: 293482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9251.739552] LNet: Removed LNI 192.168.201.133@tcp [ 9252.372540] Key type .llcrypt unregistered [ 9252.374624] Key type ._llcrypt unregistered [ 9276.510581] Key type ._llcrypt registered [ 9276.512964] Key type .llcrypt registered [ 9276.610344] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 9290.406376] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9291.546817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9291.566884] alg: No test for adler32 (adler32-zlib) [ 9292.678694] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 9292.937131] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 9294.602378] Key type lgssc registered [ 9295.757468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9339.341567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9339.354464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.555250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9340.572504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9340.649688] Lustre: lustre-MDT0000: new disk, initializing [ 9340.768659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9340.813796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9345.182814] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9354.769607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9354.882296] Lustre: 297308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9354.938538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9354.942269] Lustre: Skipped 1 previous similar message [ 9355.010192] Lustre: lustre-MDT0001: new disk, initializing [ 9355.135670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9355.204279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9355.227042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9360.017462] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9368.646823] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9373.845484] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9380.179240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9380.496316] Lustre: lustre-OST2710: new disk, initializing [ 9380.499981] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9380.505409] Lustre: 299277:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9380.585510] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9388.468848] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9390.694824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9390.699488] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9390.816124] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9398.350694] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9405.144737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9405.355664] Lustre: lustre-OST03e8: new disk, initializing [ 9405.365570] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9405.372940] Lustre: 300342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9405.499469] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9411.296470] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9412.243097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9412.256445] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9412.339258] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9418.736624] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9426.072485] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9426.288365] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9431.491781] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9431.660642] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9435.838664] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9436.057197] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9441.195147] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9441.416804] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9447.402142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9447.421783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9447.430867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9447.728071] Lustre: server umount lustre-MDT0000 complete [ 9448.424404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9448.446449] Lustre: Skipped 1 previous similar message [ 9451.540283] LustreError: 297302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758811 with bad export cookie 16543248425999339418 [ 9451.549500] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9451.553990] LustreError: 297302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9451.802971] Lustre: server umount lustre-MDT0001 complete [ 9466.377625] Lustre: server umount lustre-OST2710 complete [ 9469.736532] Lustre: 295107:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758813/real 1786758813] req@000000002a1f2e04 x1873552241398656/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786758829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9469.775154] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9469.781889] Lustre: Skipped 1 previous similar message [ 9470.684310] Lustre: server umount lustre-OST03e8 complete [ 9490.198227] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 9500.174255] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9536.769756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9537.194902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9537.236158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9537.385769] Lustre: lustre-MDT0000: new disk, initializing [ 9537.492922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9537.518904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9541.570199] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9551.949107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9552.075084] Lustre: 305589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9552.129535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9552.133953] Lustre: Skipped 1 previous similar message [ 9552.208520] Lustre: lustre-MDT0001: new disk, initializing [ 9552.318440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9552.373168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9552.388643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9557.066512] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9566.653858] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9573.599833] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9580.123259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9580.409251] Lustre: lustre-OST0000: new disk, initializing [ 9580.413562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9580.422213] Lustre: 307560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9580.527919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9586.811526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9586.838571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9587.002317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9587.616923] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9597.291816] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9602.414153] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9602.635059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9607.211165] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9607.436695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9612.791233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9612.796918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9612.806135] LustreError: Skipped 1 previous similar message [ 9612.836097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9612.841231] Lustre: Skipped 1 previous similar message [ 9615.488634] Lustre: server umount lustre-OST0000 complete [ 9623.009869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9623.018071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9623.019162] Lustre: Skipped 2 previous similar messages [ 9623.039698] Lustre: Skipped 1 previous similar message [ 9625.776408] Lustre: server umount lustre-MDT0000 complete [ 9628.134567] LustreError: 305601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9628.147072] LustreError: 305601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9629.157607] LustreError: 305580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758989 with bad export cookie 16543248425999341140 [ 9629.168959] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9629.177249] LustreError: 305580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9629.463952] Lustre: server umount lustre-MDT0001 complete [ 9637.634552] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:56:36 (1786758996) [ 9638.882723] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9640.401697] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:56:39 (1786758999) [ 9644.245188] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9668.094329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9668.349160] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9668.379357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9668.550378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9673.621416] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9683.325945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9683.556152] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9683.611818] Lustre: 311419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9689.382385] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9712.182190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9712.477634] Lustre: Denying initial registration attempt from nid 192.168.201.133@tcp, specified as failover [ 9712.490257] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9712.511464] LustreError: 312876:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9712.529301] LustreError: 312876:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9712.542406] LustreError: 312876:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9712.641609] Lustre: server umount lustre-OST0000 complete [ 9712.655699] LustreError: 312876:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9724.899536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9724.911131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9724.949073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9724.958044] Lustre: Skipped 1 previous similar message [ 9726.546619] Lustre: server umount lustre-MDT0000 complete [ 9730.017579] LustreError: 312389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9730.030569] LustreError: 312389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9730.722431] LustreError: 311414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759090 with bad export cookie 16543248425999342064 [ 9730.731212] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9730.737729] LustreError: 311414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9731.146816] Lustre: server umount lustre-MDT0001 complete [ 9740.685554] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 9742.992279] Key type lgssc unregistered [ 9743.272950] LNet: 314001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9743.285926] LNetError: 314001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9743.328812] LNet: Removed LNI 192.168.201.133@tcp [ 9744.402315] Key type .llcrypt unregistered [ 9744.406975] Key type ._llcrypt unregistered [ 9757.019748] Key type ._llcrypt registered [ 9757.022142] Key type .llcrypt registered [ 9757.112869] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9757.837831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9757.855589] alg: No test for adler32 (adler32-zlib) [ 9758.898573] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 9759.151218] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 9760.799229] Key type lgssc registered [ 9762.080975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9816.880922] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9825.930072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9825.959426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9827.343516] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9827.382749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9827.584106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9832.690800] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9842.000711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9842.070780] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9842.100236] Lustre: 316768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9842.313405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9846.829836] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9855.012115] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9860.111417] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9880.385659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9880.683441] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9880.886953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9887.445232] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9895.406045] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9900.000853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9900.009337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9900.025318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9900.522416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9900.525333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9900.552119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9900.558853] Lustre: Skipped 1 previous similar message [ 9905.633696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9910.754109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9910.763667] Lustre: Skipped 2 previous similar messages [ 9912.287514] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9912.631669] Lustre: server umount lustre-OST0000 complete [ 9918.949739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9918.956964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9918.974621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9920.997540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9923.319857] Lustre: server umount lustre-MDT0000 complete [ 9927.197427] LustreError: 316762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759287 with bad export cookie 9374542868618283644 [ 9927.200722] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9927.218781] LustreError: 316762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9927.703529] Lustre: server umount lustre-MDT0001 complete [ 9938.590608] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 9942.097594] Key type lgssc unregistered [ 9942.505887] LNet: 320360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9942.520570] LNetError: 320360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9943.593086] LNet: Removed LNI 192.168.201.133@tcp [ 9944.639651] Key type .llcrypt unregistered [ 9944.646547] Key type ._llcrypt unregistered [ 9974.112471] Key type ._llcrypt registered [ 9974.114734] Key type .llcrypt registered [ 9974.336338] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9976.275983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9976.339391] alg: No test for adler32 (adler32-zlib) [ 9977.590812] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [ 9978.039983] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 9979.887198] Key type lgssc registered [ 9981.644101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9991.727700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9991.748929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9993.242539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9997.256853] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10003.432708] LustreError: 321778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10003.460442] LustreError: 321778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10007.444521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10007.936099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10013.666701] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10021.374283] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10028.148768] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10039.025475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10039.487737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10047.498744] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10056.591073] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10078.951500] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [10079.714064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.749772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10079.757327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10079.766807] Lustre: Skipped 2 previous similar messages [10083.580360] Lustre: server umount lustre-MDT0000 complete [10084.005404] LustreError: 321775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.576686] LustreError: 321758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759447 with bad export cookie 9202788421853239901 [10087.593616] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10089.124588] LustreError: 321775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10089.156853] LustreError: 321775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10089.176977] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (stopping) [10089.963145] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10089.986683] Lustre: Skipped 2 previous similar messages [10092.960743] LustreError: 321775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10092.962862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10092.975422] LustreError: 321775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10092.979373] Lustre: Skipped 1 previous similar message [10094.025678] Lustre: server umount lustre-MDT0001 complete [10100.369760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10109.432081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10112.846441] LustreError: 323612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10112.952875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10113.006818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10117.080389] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10122.150365] LustreError: 323613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10122.156293] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10122.164478] LustreError: 323613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10126.133036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10126.562151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10131.166773] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10131.951068] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10138.824888] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10140.504447] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10145.335563] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10146.887589] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10152.427501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10152.435017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10152.448585] Lustre: Skipped 1 previous similar message [10152.460021] Lustre: Skipped 1 previous similar message [10157.271796] Lustre: server umount lustre-OST0000 complete [10162.672076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10162.675324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10162.676622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10162.688038] Lustre: Skipped 1 previous similar message [10167.127121] Lustre: server umount lustre-MDT0000 complete [10167.780263] LustreError: 325391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10167.801825] LustreError: 325391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10170.790631] LustreError: 324586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759531 with bad export cookie 9202788421853247447 [10170.796238] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10170.804709] LustreError: 324586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10171.170843] Lustre: server umount lustre-MDT0001 complete [10183.610568] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [10186.357949] Key type lgssc unregistered [10186.630716] LNet: 328072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10186.639868] LNetError: 328072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10186.667888] LNet: Removed LNI 192.168.201.133@tcp [10187.651610] Key type .llcrypt unregistered [10187.653175] Key type ._llcrypt unregistered [10222.513836] Key type ._llcrypt registered [10222.517933] Key type .llcrypt registered [10222.758881] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [10225.871828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10225.935362] alg: No test for adler32 (adler32-zlib) [10227.352291] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [10227.739606] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [10229.415132] Key type lgssc registered [10230.269130] Lustre: Echo OBD driver; http://www.lustre.org/ [10237.692840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10237.722090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10239.138937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10243.298991] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10249.709243] LustreError: 329495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10249.741404] LustreError: 329495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10252.237755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10252.600280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10257.820579] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10266.673103] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10272.518924] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10277.537821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.563048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10277.572696] Lustre: Skipped 1 previous similar message [10281.883108] Lustre: server umount lustre-MDT0000 complete [10282.977724] LustreError: 330225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10283.007826] LustreError: 330225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10285.854240] LustreError: 329476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759646 with bad export cookie 2519857879073866285 [10285.859771] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10285.863993] LustreError: 329476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10286.280286] Lustre: server umount lustre-MDT0001 complete [10300.255747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.614469] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10300.684533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10301.071708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10306.695193] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10315.722959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10315.835412] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10315.909844] Lustre: 332085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10316.313973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10321.599388] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10329.354972] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10336.874372] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10348.643562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10348.767370] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10348.772880] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10348.776734] LustreError: 333972:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10348.782446] LustreError: 333972:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10348.786731] LustreError: 333972:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10348.832602] Lustre: server umount lustre-OST0000 complete [10348.835071] LustreError: 333972:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10358.730425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10359.047699] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10359.367054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10366.129819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10366.273239] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10374.360403] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10383.110121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10383.167477] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10383.243375] Lustre: lustre-OST0001: new disk, initializing [10383.246902] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10383.251532] Lustre: 335513:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10383.343737] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10390.792737] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10392.252126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10392.261430] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10392.367299] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10400.497044] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10407.911907] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10407.935744] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10407.944517] LustreError: Skipped 1 previous similar message [10407.945209] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10408.021507] Lustre: Skipped 2 previous similar messages [10413.029571] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10413.033733] Lustre: Skipped 2 previous similar messages [10418.172055] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10418.182429] Lustre: Skipped 1 previous similar message [10419.167170] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10419.641877] Lustre: server umount lustre-OST0001 complete [10423.281877] LustreError: 336322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10423.310815] LustreError: 336322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10427.880582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10427.896902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10427.917445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10427.927933] Lustre: Skipped 1 previous similar message [10428.391106] LustreError: 336322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10428.396060] LustreError: 336322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10430.615685] Lustre: server umount lustre-OST0000 complete [10438.625211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10438.652896] Lustre: Skipped 1 previous similar message [10438.661398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10438.672017] Lustre: Skipped 3 previous similar messages [10443.018404] Lustre: server umount lustre-MDT0000 complete [10443.745213] LustreError: 332092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10443.770452] LustreError: 332092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10448.433489] LustreError: 332076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759808 with bad export cookie 2519857879073866838 [10448.448456] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10448.449821] LustreError: 332076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10448.996838] Lustre: server umount lustre-MDT0001 complete [10460.172373] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [10463.184524] Key type lgssc unregistered [10463.649985] LNet: 337481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10463.661629] LNetError: 337481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10463.693575] LNet: Removed LNI 192.168.201.133@tcp [10464.712135] Key type .llcrypt unregistered [10464.722477] Key type ._llcrypt unregistered [10537.430937] Key type ._llcrypt registered [10537.432770] Key type .llcrypt registered [10537.506167] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [10553.375570] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [10554.630222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10554.653066] alg: No test for adler32 (adler32-zlib) [10555.642453] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [10555.883382] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [10557.591215] Key type lgssc registered [10558.669186] Lustre: Echo OBD driver; http://www.lustre.org/ [10597.751969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10597.806395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10599.102083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10599.167087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10599.252543] Lustre: lustre-MDT0000: new disk, initializing [10599.388379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10599.504033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10604.985815] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10617.688827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10617.933147] Lustre: 342417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10618.083361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10618.092037] Lustre: Skipped 1 previous similar message [10618.200782] Lustre: lustre-MDT0001: new disk, initializing [10618.281919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10618.310239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10618.318756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10624.036875] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10638.103741] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10645.641376] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10653.686678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10654.035348] Lustre: lustre-OST0000: new disk, initializing [10654.043260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10654.062550] Lustre: 344389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10654.279701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10660.270300] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10660.966599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10660.981319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10661.141997] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10670.889528] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10675.522282] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10675.744591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10680.067923] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10680.324882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10686.958463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10686.961117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.971694] LustreError: Skipped 1 previous similar message [10686.976855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10686.988889] Lustre: Skipped 1 previous similar message [10687.010985] Lustre: Skipped 1 previous similar message [10688.564310] Lustre: server umount lustre-OST0000 complete [10695.140034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10695.144163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10695.151288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10697.195532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10697.196122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10698.761945] Lustre: server umount lustre-MDT0000 complete [10702.282356] LustreError: 342407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760062 with bad export cookie 9429519482114608405 [10702.288048] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10702.296112] LustreError: 342407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10702.568939] Lustre: server umount lustre-MDT0001 complete [10711.108302] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:14:30 (1786760070) [10718.966645] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:14:37 (1786760077) [10727.626066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10728.265907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10733.356764] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10733.536192] LustreError: 347144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10733.552661] LustreError: 347144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10738.660247] LustreError: 347145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.375747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10742.688584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10747.492820] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10756.931701] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10765.088336] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10772.345199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10772.822697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10779.744243] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10787.968548] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10796.241868] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10798.292846] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10800.215542] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10802.984841] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10805.304757] Lustre: Failing over lustre-MDT0000 [10805.671582] Lustre: server umount lustre-MDT0000 complete [10805.953651] LustreError: 347139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.780307] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10807.790360] Lustre: Skipped 1 previous similar message [10811.040862] LustreError: 347140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10811.063857] LustreError: 347140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10816.166757] LustreError: 347141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10816.178738] LustreError: 347141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10824.165565] Lustre: 340641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786760168/real 1786760168] req@00000000819aa000 x1873553565760640/t0(0) o400->MGC192.168.201.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786760184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10824.207983] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10824.246502] LustreError: 347144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10824.285181] LustreError: 347144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10824.580738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10834.414431] LustreError: 340637:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000031f3dfed x1873553565765760/t0(0) o250->MGC192.168.201.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10834.852691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10834.908101] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10837.681094] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10840.057580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10840.083541] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10840.169426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10841.980085] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10851.689756] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10853.211473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10856.778884] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10858.842636] Lustre: Failing over lustre-MDT0000 [10859.199602] Lustre: server umount lustre-MDT0000 complete [10860.520524] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10860.538163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10860.544158] Lustre: Skipped 1 previous similar message [10860.550064] LustreError: 349026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10860.550075] LustreError: 349026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10876.898992] Lustre: 340639:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786760220/real 1786760220] req@000000000cea04ca x1873553565782144/t0(0) o400->MGC192.168.201.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786760236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10876.933771] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10878.531631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10887.154281] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x82dc62025e1117f7 [10887.165853] Lustre: MGC192.168.201.133@tcp: Connection restored to 0@lo (at 0@lo) [10887.171706] Lustre: Skipped 2 previous similar messages [10887.470427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10887.558369] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10888.869525] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10892.255834] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10892.786914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10892.828134] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10892.874526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10900.881619] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10902.422659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10913.258268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10913.261280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10913.289257] Lustre: Skipped 3 previous similar messages [10913.290160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10913.290165] Lustre: Skipped 1 previous similar message [10914.544307] Lustre: server umount lustre-MDT0000 complete [10918.369249] LustreError: 347145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.393052] LustreError: 347145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [10918.780242] LustreError: 347124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760279 with bad export cookie 9429519482114611191 [10918.784304] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10918.798279] LustreError: 347124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10919.385926] Lustre: server umount lustre-MDT0001 complete [10934.001849] Lustre: server umount lustre-OST0000 complete [10950.876574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.129467] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10966.606508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10966.960209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10966.964702] Lustre: Skipped 1 previous similar message [10970.863552] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10977.244819] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10982.153502] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10988.302882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10994.493449] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10995.823680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11000.861674] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11006.436762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11006.448090] Lustre: Skipped 1 previous similar message [11006.459353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11006.462569] Lustre: Skipped 2 previous similar messages [11011.625728] LustreError: 355323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11011.653879] LustreError: 355323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11011.870405] Lustre: server umount lustre-OST0000 complete [11016.686558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11016.689623] Lustre: Skipped 1 previous similar message [11022.050037] Lustre: server umount lustre-MDT0000 complete [11028.224992] LustreError: 355992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760388 with bad export cookie 9429519482114611905 [11028.238026] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11028.239360] LustreError: 355992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11028.616949] Lustre: server umount lustre-MDT0001 complete [11040.425539] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11042.790398] Key type lgssc unregistered [11043.055957] LNet: 357037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11043.070847] LNetError: 357037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11044.140955] LNet: Removed LNI 192.168.201.133@tcp [11044.925428] Key type .llcrypt unregistered [11044.926847] Key type ._llcrypt unregistered [11071.238635] Key type ._llcrypt registered [11071.240826] Key type .llcrypt registered [11071.383664] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11072.793982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11072.975125] alg: No test for adler32 (adler32-zlib) [11074.319272] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [11074.686714] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11076.448239] Key type lgssc registered [11077.489562] Lustre: Echo OBD driver; http://www.lustre.org/ [11087.553416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11087.582392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11088.954655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11092.958940] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11099.107987] LustreError: 358550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.124365] LustreError: 358550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11100.888985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11101.178368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11105.147371] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11111.652790] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11115.922689] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11122.278948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11122.642766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11128.392200] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11133.944223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11135.636917] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11147.236260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11147.252586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11147.281516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11149.011294] Lustre: server umount lustre-MDT0000 complete [11149.285261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11149.287574] LustreError: 359278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.320300] LustreError: 359278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11152.597585] LustreError: 358532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760512 with bad export cookie 16301158828121605690 [11152.611527] LustreError: 358532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11152.615844] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11152.887384] Lustre: server umount lustre-MDT0001 complete [11166.184842] Lustre: server umount lustre-OST0000 complete [11180.483895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11181.022799] LustreError: 362264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11181.109276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11185.824329] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11186.145983] LustreError: 362265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.274922] LustreError: 362264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11194.053113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.345346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11197.695786] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11205.787397] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11212.346755] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11220.410561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11221.007194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11226.121474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11228.438722] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11236.630194] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11249.634195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11249.636342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11249.697433] Lustre: Skipped 3 previous similar messages [11249.708307] Lustre: Skipped 2 previous similar messages [11254.736418] Lustre: server umount lustre-MDT0000 complete [11254.759818] LustreError: 362265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11254.797756] LustreError: 362265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11261.433827] LustreError: 362245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760621 with bad export cookie 16301158828121606803 [11261.446661] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11261.448876] LustreError: 362245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11261.792333] Lustre: server umount lustre-MDT0001 complete [11277.414781] Lustre: server umount lustre-OST0000 complete [11295.175172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11295.597013] LustreError: 366177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11295.621421] LustreError: 366177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11295.676435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11300.638193] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11309.553740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11315.059812] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11321.332308] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11326.707088] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11333.072937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11333.375323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11333.379743] Lustre: Skipped 1 previous similar message [11339.827208] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11340.603897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11348.011747] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11355.676249] Lustre: server umount lustre-OST0000 complete [11361.250254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11361.252244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11361.269748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11361.290784] Lustre: Skipped 1 previous similar message [11366.372387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11366.381868] Lustre: Skipped 2 previous similar messages [11369.527081] Lustre: server umount lustre-MDT0000 complete [11371.491146] LustreError: 366178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11371.509585] LustreError: 366178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11372.512264] Lustre: 357864:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786760716/real 1786760716] req@00000000988731b6 x1873554108930432/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786760732 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11372.514153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11372.546026] Lustre: 357864:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11374.117954] LustreError: 366913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760734 with bad export cookie 16301158828121607825 [11374.138936] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11374.139388] LustreError: 366913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11374.521288] Lustre: server umount lustre-MDT0001 complete [11386.065275] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11389.177669] Key type lgssc unregistered [11389.569582] LNet: 369715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11389.583388] LNetError: 369715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11389.604811] LNet: Removed LNI 192.168.201.133@tcp [11390.321527] Key type .llcrypt unregistered [11390.323470] Key type ._llcrypt unregistered [11413.880624] Key type ._llcrypt registered [11413.883599] Key type .llcrypt registered [11414.014810] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11414.918354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11415.059684] alg: No test for adler32 (adler32-zlib) [11416.299799] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [11416.550800] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11418.191654] Key type lgssc registered [11419.364886] Lustre: Echo OBD driver; http://www.lustre.org/ [11427.383514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11427.411937] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11427.426089] LustreError: 371156:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11427.455729] LustreError: 371156:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11427.470731] LustreError: 371156:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11427.482155] LustreError: 371156:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11427.495638] LustreError: 371156:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11435.885493] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11435.891507] LustreError: 371553:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11435.915984] LustreError: 371553:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11435.920806] LustreError: 371553:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11435.924799] LustreError: 371553:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11435.930046] LustreError: 371553:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11455.487941] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11458.608571] Key type lgssc unregistered [11459.019558] LNet: 372560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11459.025729] LNetError: 372560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11459.049370] LNet: Removed LNI 192.168.201.133@tcp [11459.863829] Key type .llcrypt unregistered [11459.870047] Key type ._llcrypt unregistered [11480.469659] Key type ._llcrypt registered [11480.471329] Key type .llcrypt registered [11480.591649] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [11498.277608] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11499.317855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11499.421193] alg: No test for adler32 (adler32-zlib) [11500.835325] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [11501.233670] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11503.039165] Key type lgssc registered [11504.194056] Lustre: Echo OBD driver; http://www.lustre.org/ [11545.042167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11545.064766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11546.510838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11546.590180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11546.815611] Lustre: lustre-MDT0000: new disk, initializing [11546.977976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11547.002813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11554.372457] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11568.496433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11568.718478] Lustre: 375716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11568.828514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11568.836382] Lustre: Skipped 1 previous similar message [11568.977539] Lustre: lustre-MDT0001: new disk, initializing [11569.221881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11569.322130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11569.354300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11573.959728] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11583.664705] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11589.553697] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11597.069097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11597.255909] Lustre: lustre-OST0000: new disk, initializing [11597.258719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11597.262756] Lustre: 377690:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11597.351268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11604.311385] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11606.638384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11606.659156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11606.822166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11615.350978] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11620.995830] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11621.259736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11627.068569] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11627.501882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11632.611034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11632.616441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11632.637899] Lustre: Skipped 1 previous similar message [11632.663388] Lustre: Skipped 1 previous similar message [11637.737868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11637.744053] Lustre: Skipped 1 previous similar message [11642.854445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11642.860218] Lustre: Skipped 1 previous similar message [11643.871194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11644.124360] Lustre: server umount lustre-OST0000 complete [11647.968478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11647.974659] Lustre: Skipped 1 previous similar message [11647.976570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11647.978969] Lustre: Skipped 1 previous similar message [11653.091053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11653.100925] Lustre: Skipped 1 previous similar message [11662.303263] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11663.044151] Lustre: server umount lustre-MDT0000 complete [11667.388170] LustreError: 375708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761027 with bad export cookie 16587821473334794194 [11667.392914] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11667.398671] LustreError: 375708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11667.976499] Lustre: server umount lustre-MDT0001 complete [11677.165034] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11678.772530] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:30:37 (1786761037) [11686.598118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11687.081977] LustreError: 380153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11687.105680] LustreError: 380153:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11687.169035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11692.173576] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11692.513112] LustreError: 380154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11697.639076] LustreError: 380153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11702.653332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11702.754858] LustreError: 380154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11703.305878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11711.813262] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11723.614349] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11732.343947] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11740.965630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11741.218261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11750.985977] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11759.715540] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11769.016057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11769.250405] Lustre: lustre-OST0001: new disk, initializing [11769.254673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11769.268541] Lustre: 382997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11769.391545] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11777.529141] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11778.778558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11778.794218] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11778.920809] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11788.397492] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11794.416996] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11794.420972] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.435353] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11794.450420] Lustre: Skipped 1 previous similar message [11794.459398] Lustre: Skipped 2 previous similar messages [11798.465947] Lustre: server umount lustre-OST0001 complete [11799.540423] LustreError: 382601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11799.565078] LustreError: 382601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11809.766679] LustreError: 381980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11809.795476] LustreError: 381980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11819.489310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11819.503641] LustreError: Skipped 1 previous similar message [11819.506215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11819.524678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11819.527361] Lustre: Skipped 2 previous similar messages [11828.198420] LustreError: 381978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11828.225754] LustreError: 381978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11830.239795] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11830.369594] Lustre: server umount lustre-OST0000 complete [11836.386394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11836.399893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11836.418336] Lustre: Skipped 1 previous similar message [11840.345914] Lustre: server umount lustre-MDT0000 complete [11844.887142] LustreError: 380133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761205 with bad export cookie 16587821473334795118 [11844.892867] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11844.902690] LustreError: 380133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11851.352544] Lustre: server umount lustre-MDT0001 complete [11864.647753] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11867.818540] Key type lgssc unregistered [11868.166377] LNet: 384986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11868.176276] LNetError: 384986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11868.191423] LNet: Removed LNI 192.168.201.133@tcp [11869.269938] Key type .llcrypt unregistered [11869.273499] Key type ._llcrypt unregistered [11928.648306] Key type ._llcrypt registered [11928.654635] Key type .llcrypt registered [11928.987350] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11931.354206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11931.645895] alg: No test for adler32 (adler32-zlib) [11933.132882] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [11933.659033] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11935.520136] Key type lgssc registered [11937.092312] Lustre: Echo OBD driver; http://www.lustre.org/ [11946.593803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11946.609669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11947.980253] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11948.023442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11948.193592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11954.127856] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11963.939830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11964.088496] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11964.127944] Lustre: 387788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11964.283793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11970.685581] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11982.953727] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11992.397659] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11999.714137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11999.733914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12002.503664] Lustre: server umount lustre-MDT0000 complete [12006.982845] LustreError: 387778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761367 with bad export cookie 407627457274285957 [12006.994697] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12007.006401] LustreError: 387778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12007.477636] Lustre: server umount lustre-MDT0001 complete [12017.694221] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:36:16 (1786761376) [12026.662412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12027.295686] LustreError: 390305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12027.316982] LustreError: 390305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12027.397057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12031.670645] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12032.484675] LustreError: 390306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12037.606277] LustreError: 390305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12040.180682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12040.552566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12044.586931] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12052.611343] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12058.191727] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12065.840423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12066.067222] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12066.291075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12073.400492] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12082.034334] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12089.768488] Lustre: Permanently deactivating lustre-OST0000 [12089.784784] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12100.215652] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12101.818352] LustreError: 393295:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12101.840589] LustreError: 393295:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12101.848268] LustreError: 393295:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12105.318357] LustreError: 393344:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12105.325423] LustreError: 393344:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12113.598320] Lustre: server umount lustre-OST0000 complete [12117.260920] LustreError: 393593:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12117.268327] LustreError: 393593:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12122.593046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12122.599841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12122.608914] Lustre: Skipped 1 previous similar message [12122.661049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12122.669939] Lustre: Skipped 1 previous similar message [12127.713813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12130.000354] Lustre: server umount lustre-MDT0000 complete [12132.832391] LustreError: 390306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12132.852682] LustreError: 390306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12134.210237] LustreError: 390286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761494 with bad export cookie 407627457274286538 [12134.214359] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12134.221336] LustreError: 390286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12144.831951] Lustre: server umount lustre-MDT0001 complete [12155.168088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12160.906414] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12164.012720] LustreError: 394754:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12171.927809] Lustre: 395093:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.133@tcp [12174.107440] Lustre: server umount MGS complete [12186.874679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12187.344290] LustreError: 395817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12187.363897] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12187.373183] Lustre: Skipped 1 previous similar message [12187.419777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12192.620790] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12197.856407] LustreError: 395818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12197.868954] LustreError: 395818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12202.669517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12202.963334] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12207.755829] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12217.096263] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12224.564534] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12227.182581] Lustre: Permanently reactivating lustre-OST0000 [12227.201917] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12227.205926] Lustre: Skipped 2 previous similar messages [12256.223223] LustreError: 397596:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12263.957124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12264.361827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12264.371642] Lustre: Skipped 1 previous similar message [12271.118834] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12279.466665] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12286.725312] Lustre: server umount lustre-OST0000 complete [12292.065180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12292.071245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12292.090321] Lustre: Skipped 2 previous similar messages [12292.111799] Lustre: Skipped 2 previous similar messages [12297.208106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12297.216046] Lustre: Skipped 1 previous similar message [12303.327364] Lustre: 387108:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786761647/real 1786761647] req@00000000cd7043f6 x1873555009671808/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786761663 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12303.332187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12303.354409] Lustre: 387108:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12305.378065] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12305.498416] Lustre: server umount lustre-MDT0000 complete [12308.322987] Lustre: 387110:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786761652/real 1786761652] req@00000000ad6c4c65 x1873555009673472/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786761668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12308.449824] LustreError: 396552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.474899] LustreError: 396552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12309.836924] LustreError: 395798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761670 with bad export cookie 407627457274287588 [12309.839032] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12309.846614] LustreError: 395798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12310.170939] Lustre: server umount lustre-MDT0001 complete [12321.731133] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12325.074028] Key type lgssc unregistered [12325.437336] LNet: 400140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12325.441125] LNetError: 400140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12325.465084] LNet: Removed LNI 192.168.201.133@tcp [12326.626063] Key type .llcrypt unregistered [12326.628872] Key type ._llcrypt unregistered [12347.885645] Key type ._llcrypt registered [12347.888728] Key type .llcrypt registered [12348.034809] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [12363.175929] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [12364.377778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12364.396307] alg: No test for adler32 (adler32-zlib) [12365.522941] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [12365.867862] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [12367.559231] Key type lgssc registered [12368.553809] Lustre: Echo OBD driver; http://www.lustre.org/ [12406.628122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12406.665385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12407.983627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12408.087903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12408.211543] Lustre: lustre-MDT0000: new disk, initializing [12408.335394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12408.376068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12413.517899] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12425.588607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12425.782792] Lustre: 403304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12425.874554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12425.882043] Lustre: Skipped 1 previous similar message [12425.996343] Lustre: lustre-MDT0001: new disk, initializing [12426.117876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12426.172736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12426.190494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12431.687969] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12443.802947] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12451.029754] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12460.118907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12460.345660] Lustre: lustre-OST0000: new disk, initializing [12460.349269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12460.354320] Lustre: 405276:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12460.430692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12466.344969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12466.363068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12466.552584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12468.860527] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12479.658148] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12484.954933] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12485.258485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12490.420848] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12490.804324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12497.383033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12497.386334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12497.392169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12497.408347] Lustre: Skipped 1 previous similar message [12499.312866] Lustre: server umount lustre-OST0000 complete [12507.620471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12507.647813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12507.659957] Lustre: Skipped 1 previous similar message [12509.854610] Lustre: server umount lustre-MDT0000 complete [12514.012549] LustreError: 403297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761874 with bad export cookie 10812324500934906115 [12514.020100] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12514.024315] LustreError: 403297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12514.269840] Lustre: server umount lustre-MDT0001 complete [12524.329265] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12526.963094] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:44:44 (1786761884) [12535.837819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12536.671996] LustreError: 407739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12536.700991] LustreError: 407739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12536.809996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12541.923320] LustreError: 407740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12542.310714] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12546.017212] LustreError: 407739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12551.136736] LustreError: 407740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12551.965068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12552.344148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12558.000484] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12565.879670] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12572.315482] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12580.332187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12580.726889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12587.727659] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12595.987848] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12598.148861] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12601.032559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12612.578841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12612.584135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12612.590897] Lustre: Skipped 1 previous similar message [12612.595892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12613.982162] Lustre: server umount lustre-OST0000 complete [12618.725701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12618.730964] LustreError: Skipped 1 previous similar message [12618.741145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12618.754164] Lustre: Skipped 1 previous similar message [12618.757881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12618.763941] Lustre: Skipped 1 previous similar message [12622.820552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12622.821057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12622.852029] Lustre: Skipped 1 previous similar message [12624.699554] Lustre: server umount lustre-MDT0000 complete [12627.941882] LustreError: 407734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12627.955811] LustreError: 407734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12628.282692] LustreError: 407721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761988 with bad export cookie 10812324500934907039 [12628.293438] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12628.297833] LustreError: 407721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12628.644576] Lustre: server umount lustre-MDT0001 complete [12640.320255] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12643.688620] Key type lgssc unregistered [12644.038875] LNet: 411469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12644.045373] LNetError: 411469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12644.067317] LNet: Removed LNI 192.168.201.133@tcp [12644.838774] Key type .llcrypt unregistered [12644.841682] Key type ._llcrypt unregistered [12671.218764] Key type ._llcrypt registered [12671.221233] Key type .llcrypt registered [12671.330447] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12671.721548] Key type .llcrypt unregistered [12671.727305] Key type ._llcrypt unregistered [12690.393244] Key type ._llcrypt registered [12690.396175] Key type .llcrypt registered [12690.513936] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [12692.034711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12692.294557] alg: No test for adler32 (adler32-zlib) [12693.366569] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [12693.604452] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [12695.255995] Key type lgssc registered [12696.910730] Lustre: Echo OBD driver; http://www.lustre.org/ [12705.903871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12705.926346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12707.379416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12712.725825] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12717.544260] LustreError: 413688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12717.570282] LustreError: 413688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12722.674711] LustreError: 413689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12723.062394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12723.699936] LustreError: 413688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12723.735800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12728.804855] LustreError: 413689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12728.837832] LustreError: 413689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12731.709720] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12731.970721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12737.006784] LustreError: 413688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12737.049045] LustreError: 413688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12739.550378] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12747.240200] LustreError: 413689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12747.272482] LustreError: 413689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12748.837834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12749.294281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12754.655434] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12767.719843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12767.742718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12767.808024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12769.776114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12769.778148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12769.790066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12772.545849] Lustre: server umount lustre-OST0000 complete [12780.006043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12780.010124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12780.040162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12780.040175] Lustre: Skipped 1 previous similar message [12783.590471] Lustre: server umount lustre-MDT0000 complete [12785.120235] LustreError: 413688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12785.136843] LustreError: 413688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12787.518337] LustreError: 413668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762147 with bad export cookie 17048472752949610253 [12787.521266] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12787.526024] LustreError: 413668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12787.911985] Lustre: server umount lustre-MDT0001 complete [12798.666411] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12802.104391] Key type lgssc unregistered [12802.441811] LNet: 416843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12802.445816] LNetError: 416843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12803.498426] LNet: Removed LNI 192.168.201.133@tcp [12804.636184] Key type .llcrypt unregistered [12804.640156] Key type ._llcrypt unregistered [12830.734827] Key type ._llcrypt registered [12830.736799] Key type .llcrypt registered [12830.869804] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [12832.719896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12832.800858] alg: No test for adler32 (adler32-zlib) [12833.943873] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [12834.292799] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [12836.111264] Key type lgssc registered [12837.624857] Lustre: Echo OBD driver; http://www.lustre.org/ [12846.878390] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12846.912438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12862.559572] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12867.680049] LustreError: 418236:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [12894.433338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12901.750249] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12910.982183] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12920.473682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12921.241800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12921.367620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12926.433243] LustreError: 418262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12926.463312] LustreError: 418262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12927.094608] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12929.523127] LustreError: 418261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12929.572514] LustreError: 418261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12934.626770] LustreError: 418262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12934.684269] LustreError: 418262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12937.177046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12937.644611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12941.539418] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12951.520537] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12957.724442] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12967.393305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12967.405512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12967.426141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12968.433016] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12968.441667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12968.445826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12968.483488] Lustre: Skipped 1 previous similar message [12972.447658] Lustre: server umount lustre-OST0000 complete [12978.669347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12978.670197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12978.697720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12983.156488] Lustre: server umount lustre-MDT0000 complete [12983.785457] LustreError: 419273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12983.806994] LustreError: 419273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12987.889860] LustreError: 419272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12987.918605] LustreError: 419272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12988.429995] LustreError: 418242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762348 with bad export cookie 15529745040929777527 [12988.439747] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12988.445044] LustreError: 418242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12989.249055] Lustre: server umount lustre-MDT0001 complete [13002.002096] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13005.294349] Key type lgssc unregistered [13005.631461] LNet: 421816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13005.646178] LNetError: 421816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13005.669842] LNet: Removed LNI 192.168.201.133@tcp [13006.578142] Key type .llcrypt unregistered [13006.580228] Key type ._llcrypt unregistered [13034.631614] Key type ._llcrypt registered [13034.634547] Key type .llcrypt registered [13034.767727] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13036.415137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13036.512385] alg: No test for adler32 (adler32-zlib) [13037.557134] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [13037.811861] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [13039.625847] Key type lgssc registered [13041.713076] Lustre: Echo OBD driver; http://www.lustre.org/ [13051.702097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13051.731523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13053.400968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13058.809692] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13063.650418] LustreError: 423236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13063.680626] LustreError: 423236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13068.772743] LustreError: 423235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13069.182626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13069.720178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13075.356712] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13085.091736] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13091.004479] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13098.145284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13098.410346] LustreError: 425061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13098.718804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13104.126247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13106.575657] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13115.481064] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13120.994089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13121.012824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13121.046696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13123.270353] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [13124.580768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13124.590454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13124.614611] Lustre: Skipped 2 previous similar messages [13127.126293] Lustre: server umount lustre-MDT0000 complete [13128.370439] LustreError: 423232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13129.853432] LustreError: 423215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.33@tcp arrived at 1786762490 with bad export cookie 14500123215183886754 [13129.874611] LustreError: 423215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13132.509163] LustreError: 423216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762492 with bad export cookie 14500123215183885858 [13132.527122] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13132.527978] LustreError: 423216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13134.815799] LustreError: 423236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13134.817725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13134.833587] LustreError: 423236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13134.860825] Lustre: Skipped 1 previous similar message [13134.865824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13138.787353] Lustre: server umount lustre-OST0000 complete [13146.887939] Lustre: server umount lustre-MDT0001 complete [13160.808620] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13163.725669] Key type lgssc unregistered [13164.039196] LNet: 426926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13164.050256] LNetError: 426926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13164.077707] LNet: Removed LNI 192.168.201.133@tcp [13165.111262] Key type .llcrypt unregistered [13165.113084] Key type ._llcrypt unregistered [13194.397879] Key type ._llcrypt registered [13194.404734] Key type .llcrypt registered [13194.572781] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13196.111784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13196.308354] alg: No test for adler32 (adler32-zlib) [13197.392432] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [13197.592445] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [13199.271296] Key type lgssc registered [13200.872165] Lustre: Echo OBD driver; http://www.lustre.org/ [13211.842631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13211.876017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13213.662123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13219.656117] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13222.882657] LustreError: 428345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13222.906793] LustreError: 428345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13228.017467] LustreError: 428346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13228.719785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13229.238725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13234.331589] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13243.028567] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13249.799957] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13258.808128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13259.266119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13264.378848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13265.915652] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13275.089254] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13280.744947] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13280.759521] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13280.785770] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13281.988409] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (stopping) [13282.597615] Lustre: server umount lustre-MDT0001 complete [13284.844896] LustreError: 428345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13284.852673] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13289.967639] LustreError: 428345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13290.007897] LustreError: 428345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13295.076069] LustreError: 428346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13295.077481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13295.140697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13295.169927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13296.951741] Lustre: server umount lustre-OST0000 complete [13302.549842] Lustre: server umount lustre-MDT0000 complete [13317.658876] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13320.930878] Key type lgssc unregistered [13321.483870] LNet: 432036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13321.489810] LNetError: 432036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13321.519258] LNet: Removed LNI 192.168.201.133@tcp [13322.843240] Key type .llcrypt unregistered [13322.845856] Key type ._llcrypt unregistered [13352.828979] Key type ._llcrypt registered [13352.831964] Key type .llcrypt registered [13352.961506] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [13368.568285] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13369.733368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13369.789032] alg: No test for adler32 (adler32-zlib) [13370.849951] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [13371.113447] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [13372.791232] Key type lgssc registered [13374.245491] Lustre: Echo OBD driver; http://www.lustre.org/ [13417.227552] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13434.204126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13434.228046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13435.566966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13435.607612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13435.710724] Lustre: lustre-MDT0000: new disk, initializing [13435.846251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13435.928070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13441.039910] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13458.554968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13458.961790] Lustre: lustre-OST0000: new disk, initializing [13458.968931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13458.976891] Lustre: Skipped 1 previous similar message [13458.996277] Lustre: 437075:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13459.098861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13466.699140] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13467.264725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13467.272983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13467.306993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13476.854381] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13494.585857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13494.727768] Lustre: 436057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13494.844320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13494.931746] Lustre: lustre-MDT0001: new disk, initializing [13495.034548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13495.075413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13495.091546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13500.572186] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13511.555282] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13511.743941] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13517.596938] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13517.796069] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13525.720742] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (stopping) [13527.523506] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13527.525048] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13527.557226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13527.564115] Lustre: Skipped 1 previous similar message [13529.491279] Lustre: server umount lustre-MDT0001 complete [13530.807134] LustreError: 436063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13530.828427] LustreError: 436063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13532.651384] LustreError: 436063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13532.686584] LustreError: 436063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13537.774302] LustreError: 436063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13537.804362] LustreError: 436063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13542.883044] LustreError: 436069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13548.004559] LustreError: 436063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13548.020455] LustreError: 436063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13549.541960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13549.554858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13549.573015] Lustre: Skipped 1 previous similar message [13549.608305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13553.141379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13554.797555] Lustre: server umount lustre-OST0000 complete [13558.245388] LustreError: 438714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13558.264753] LustreError: 438714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13565.570049] Lustre: server umount lustre-MDT0000 complete [13583.299644] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13586.195943] Key type lgssc unregistered [13586.513884] LNet: 440841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13586.517208] LNetError: 440841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13587.557830] LNet: Removed LNI 192.168.201.133@tcp [13588.492485] Key type .llcrypt unregistered [13588.494644] Key type ._llcrypt unregistered [13629.691461] Key type ._llcrypt registered [13629.694731] Key type .llcrypt registered [13629.796249] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13630.079495] Key type .llcrypt unregistered [13630.082270] Key type ._llcrypt unregistered [13648.689507] Key type ._llcrypt registered [13648.691579] Key type .llcrypt registered [13648.791694] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13650.147917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13650.266409] alg: No test for adler32 (adler32-zlib) [13651.285546] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [13651.506496] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [13653.191189] Key type lgssc registered [13654.227201] Lustre: Echo OBD driver; http://www.lustre.org/ [13687.132854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13687.161832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13688.484424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13688.525872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13688.669119] Lustre: lustre-MDT0000: new disk, initializing [13688.831611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13688.864320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13693.402426] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13705.168727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13705.308970] Lustre: 444226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13705.385302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13705.389734] Lustre: Skipped 1 previous similar message [13705.484746] Lustre: lustre-MDT0001: new disk, initializing [13705.611247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13705.659746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13705.675428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13710.468720] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13719.446993] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13725.837967] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13732.753703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13732.989440] Lustre: lustre-OST0000: new disk, initializing [13732.995627] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13733.001629] Lustre: 446196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13733.083825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13738.971986] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13742.663152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13742.680117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13742.835307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13748.292751] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13756.900624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13756.907947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13756.940133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13758.440947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13758.442577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13758.452831] Lustre: Skipped 1 previous similar message [13758.459799] Lustre: Skipped 1 previous similar message [13762.222618] Lustre: server umount lustre-MDT0000 complete [13766.855953] LustreError: 445032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763127 with bad export cookie 9073242522966235844 [13766.863895] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13766.870227] LustreError: 445032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13767.168090] Lustre: server umount lustre-MDT0001 complete [13782.277178] Lustre: server umount lustre-OST0000 complete [13796.397764] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:05:54 (1786763154) [13807.876555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13808.789341] LustreError: 448225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13808.824822] LustreError: 448225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13808.922185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13813.893950] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13814.247681] LustreError: 448226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13819.368306] LustreError: 448225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13824.481037] LustreError: 448226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13824.570806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13825.028803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13831.255298] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13840.065468] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13846.460562] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13867.134305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13867.337825] Lustre: Found index 0 for lustre-OST0000, updating log [13867.571209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13873.779824] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13879.862452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13881.789227] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13890.024081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13890.028884] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13890.032298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13890.046761] Lustre: Skipped 1 previous similar message [13890.059418] Lustre: Skipped 2 previous similar messages [13893.585895] Lustre: server umount lustre-OST0000 complete [13912.033290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13912.041174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13912.049232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13912.057857] Lustre: Skipped 2 previous similar messages [13915.964076] Lustre: server umount lustre-MDT0000 complete [13917.665292] LustreError: 448961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13917.691828] LustreError: 448961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13919.508524] LustreError: 451075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763279 with bad export cookie 9073242522966237279 [13919.516224] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13919.863212] Lustre: server umount lustre-MDT0001 complete [13928.031779] LNet: 1 local NIs in recovery (showing 1): 192.168.201.133@tcp [13928.947355] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:08:07 (1786763287) [14549.719195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14550.273794] LustreError: 454642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14550.306140] LustreError: 454642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14550.400444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14554.460351] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [14555.630046] LustreError: 454643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14560.739958] LustreError: 454642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14563.891604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14564.228535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14570.609498] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [14579.767265] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14587.131249] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14606.047285] LNet: 1 local NIs in recovery (showing 1): 192.168.201.133@tcp [14607.023571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14607.338451] Lustre: Found index 0 for lustre-OST0000, updating log [14607.751142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14614.223272] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [14622.533742] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14625.698647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14625.720561] Lustre: Skipped 1 previous similar message [14625.726155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14625.730599] Lustre: Skipped 1 previous similar message [14630.886690] Lustre: server umount lustre-OST0000 complete [14636.002413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14636.009765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14636.029315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14637.090410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14637.097685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14639.201097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14639.215306] Lustre: Skipped 4 previous similar messages [14641.815656] Lustre: server umount lustre-MDT0000 complete [14642.276079] LustreError: 455371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14642.296688] LustreError: 455371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14645.721462] LustreError: 456495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786764005 with bad export cookie 9073242522966238364 [14645.723401] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14645.727981] LustreError: 456495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14646.137912] Lustre: server umount lustre-MDT0001 complete [14666.474547] LNet: 1 local NIs in recovery (showing 1): 192.168.201.133@tcp [14679.448885] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [14691.541524] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [14736.045588] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:21:34 (1786764094) [14759.043273] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [14762.207560] Key type lgssc unregistered [14762.446342] LNet: 461259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14762.450533] LNetError: 461259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14869.991071] LNet: Removed LNI 192.168.201.133@tcp [14871.218090] Key type .llcrypt unregistered [14871.219857] Key type ._llcrypt unregistered [14889.347218] Key type ._llcrypt registered [14889.357320] Key type .llcrypt registered [14889.548733] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [14890.962596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14891.245839] alg: No test for adler32 (adler32-zlib) [14892.314596] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [14892.511783] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [14894.207751] Key type lgssc registered [14895.362788] Lustre: Echo OBD driver; http://www.lustre.org/ [14899.764337] LNet: Added LNI 192.168.201.133@tcp100 [8/256/0/180] [14900.727111] LNet: Added LNI 192.168.201.133@tcp101 [8/256/0/180] [14901.795616] LNet: Added LNI 192.168.201.133@tcp102 [8/256/0/180] [14902.789994] LNet: Added LNI 192.168.201.133@tcp103 [8/256/0/180] [14903.828047] LNet: Added LNI 192.168.201.133@tcp104 [8/256/0/180] [14904.707743] LNet: Added LNI 192.168.201.133@tcp105 [8/256/0/180] [14905.589697] LNet: Added LNI 192.168.201.133@tcp106 [8/256/0/180] [14906.705965] LNet: Added LNI 192.168.201.133@tcp107 [8/256/0/180] [14907.717441] LNet: Added LNI 192.168.201.133@tcp108 [8/256/0/180] [14909.004683] LNet: Added LNI 192.168.201.133@tcp109 [8/256/0/180] [14910.097255] LNet: Added LNI 192.168.201.133@tcp110 [8/256/0/180] [14911.087479] LNet: Added LNI 192.168.201.133@tcp111 [8/256/0/180] [14911.942463] LNet: Added LNI 192.168.201.133@tcp112 [8/256/0/180] [14912.775800] LNet: Added LNI 192.168.201.133@tcp113 [8/256/0/180] [14913.622600] LNet: Added LNI 192.168.201.133@tcp114 [8/256/0/180] [14914.431621] LNet: Added LNI 192.168.201.133@tcp115 [8/256/0/180] [14915.342548] LNet: Added LNI 192.168.201.133@tcp116 [8/256/0/180] [14916.313526] LNet: Added LNI 192.168.201.133@tcp117 [8/256/0/180] [14917.077085] LNet: Added LNI 192.168.201.133@tcp118 [8/256/0/180] [14917.890421] LNet: Added LNI 192.168.201.133@tcp119 [8/256/0/180] [14918.620322] LNet: Added LNI 192.168.201.133@tcp120 [8/256/0/180] [14919.560444] LNet: Added LNI 192.168.201.133@tcp121 [8/256/0/180] [14920.467375] LNet: Added LNI 192.168.201.133@tcp122 [8/256/0/180] [14921.355525] LNet: Added LNI 192.168.201.133@tcp123 [8/256/0/180] [14922.191888] LNet: Added LNI 192.168.201.133@tcp124 [8/256/0/180] [14923.176929] LNet: Added LNI 192.168.201.133@tcp125 [8/256/0/180] [14924.073676] LNet: Added LNI 192.168.201.133@tcp126 [8/256/0/180] [14925.026327] LNet: Added LNI 192.168.201.133@tcp127 [8/256/0/180] [14926.118275] LNet: Added LNI 192.168.201.133@tcp128 [8/256/0/180] [14927.087882] LNet: Added LNI 192.168.201.133@tcp129 [8/256/0/180] [14927.911512] LNet: Added LNI 192.168.201.133@tcp130 [8/256/0/180] [14928.781348] LNet: Added LNI 192.168.201.133@tcp131 [8/256/0/180] [14929.977556] LNet: Added LNI 192.168.201.133@tcp132 [8/256/0/180] [14930.988527] LNet: Added LNI 192.168.201.133@tcp133 [8/256/0/180] [14932.009460] LNet: Added LNI 192.168.201.133@tcp134 [8/256/0/180] [14932.914087] LNet: Added LNI 192.168.201.133@tcp135 [8/256/0/180] [14938.997641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14939.009744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14940.109341] Lustre: 464289:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.133@tcp: can't add NID 192.168.201.133@tcp130: rc = -75 [14940.172324] LNetError: 464310:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14940.179167] LNetError: Protocol error connecting to 192.168.201.133@tcp114 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14940.206654] LNetError: Skipped 8 previous similar messages [14940.286803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14940.311934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14940.441326] Lustre: lustre-MDT0000: new disk, initializing [14940.519157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14940.571965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14940.644622] LNet: 6 local NIs in recovery (showing 6): 192.168.201.133@tcp121, 192.168.201.133@tcp122, 192.168.201.133@tcp124, 192.168.201.133@tcp125, 192.168.201.133@tcp126, 192.168.201.133@tcp127 [14941.216831] LNetError: 461748:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14941.232133] LNetError: 461748:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [14941.241525] LNetError: Protocol error connecting to 192.168.201.133@tcp104 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14941.250029] LNetError: Skipped 2 previous similar messages [14943.776521] LNetError: 461750:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14943.809537] LNetError: 461750:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [14943.822262] LNetError: Protocol error connecting to 192.168.201.133@tcp121 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14943.830897] LNetError: Skipped 6 previous similar messages [14944.596728] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [14947.877913] LNetError: 464317:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14947.905238] LNetError: Protocol error connecting to 192.168.201.133@tcp121 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14955.041202] LNetError: 464310:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14955.059272] LNetError: Protocol error connecting to 192.168.201.133@tcp121 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14955.350873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14955.537934] Lustre: 464307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14955.631178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14955.634610] Lustre: Skipped 1 previous similar message [14955.698971] Lustre: lustre-MDT0001: new disk, initializing [14955.743162] Lustre: 465106:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.133@tcp: can't add NID 192.168.201.133@tcp130: rc = -75 [14955.750663] Lustre: 465106:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14955.852753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14955.899267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14955.917375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14961.267332] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [14970.338669] LNetError: 461750:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14970.349287] LNetError: 461750:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 30 previous similar messages [14970.356813] LNetError: Protocol error connecting to 192.168.201.133@tcp123 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14970.368799] LNetError: Skipped 30 previous similar messages [14970.658812] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14976.466931] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14983.615355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14983.941495] Lustre: lustre-OST0000: new disk, initializing [14983.945180] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14983.956602] Lustre: 466281:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14984.051451] Lustre: 466286:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.133@tcp: can't add NID 192.168.201.133@tcp130: rc = -75 [14984.055100] Lustre: 466286:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14984.088584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14986.815283] LNetError: 461750:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14986.820571] LNetError: 461750:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [14986.824471] LNetError: Protocol error connecting to 192.168.201.133@tcp129 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14986.829159] LNetError: Skipped 19 previous similar messages [14989.396409] Lustre: 466574:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.133@tcp: can't add NID 192.168.201.133@tcp130: rc = -75 [14989.412502] Lustre: 466574:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14989.499381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14989.507845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14989.656756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14990.269475] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15000.159143] LNet: 4 local NIs in recovery (showing 4): 192.168.201.133@tcp129, 192.168.201.133@tcp133, 192.168.201.133@tcp134, 192.168.201.133@tcp121 [15004.435918] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15015.393225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15015.393478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15015.411978] LustreError: Skipped 1 previous similar message [15015.415030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15015.481286] Lustre: Skipped 1 previous similar message [15019.553686] LNetError: 464310:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [15019.561438] LNetError: 464310:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 33 previous similar messages [15019.566777] LNetError: Protocol error connecting to 192.168.201.133@tcp120 at host 192.168.201.133:988: is it running a compatible version of Lustre? [15019.573932] LNetError: Skipped 30 previous similar messages [15020.520968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15020.531694] Lustre: Skipped 1 previous similar message [15020.932029] Lustre: server umount lustre-OST0000 complete [15025.639876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15025.644748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15025.652163] Lustre: Skipped 1 previous similar message [15025.656444] Lustre: Skipped 1 previous similar message [15030.695751] Lustre: server umount lustre-MDT0000 complete [15034.535299] LustreError: 464297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786764394 with bad export cookie 4549971194237101233 [15034.542782] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15034.553716] LustreError: 464297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15034.844306] Lustre: server umount lustre-MDT0001 complete [15046.313242] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [15048.759473] Key type lgssc unregistered [15049.061723] LNet: 468197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15049.067106] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15049.073897] LNet: Removed LNI 192.168.201.133@tcp [15050.079142] LNet: Removed LNI 192.168.201.133@tcp100 [15050.089136] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15050.095469] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15050.110185] LNet: Removed LNI 192.168.201.133@tcp101 [15051.167174] LNet: Removed LNI 192.168.201.133@tcp102 [15051.169037] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15051.171477] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15051.175755] LNet: Removed LNI 192.168.201.133@tcp103 [15052.194397] LNet: Removed LNI 192.168.201.133@tcp104 [15053.280366] LNet: Removed LNI 192.168.201.133@tcp105 [15053.282781] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15053.287484] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15054.303191] LNet: Removed LNI 192.168.201.133@tcp106 [15055.327187] LNet: Removed LNI 192.168.201.133@tcp107 [15056.351967] LNet: Removed LNI 192.168.201.133@tcp108 [15057.376417] LNet: Removed LNI 192.168.201.133@tcp109 [15057.383281] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15057.392789] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [15057.404115] LNet: Removed LNI 192.168.201.133@tcp110 [15058.465127] LNet: Removed LNI 192.168.201.133@tcp111 [15059.489024] LNet: Removed LNI 192.168.201.133@tcp112 [15060.511261] LNet: Removed LNI 192.168.201.133@tcp113 [15061.535306] LNet: Removed LNI 192.168.201.133@tcp114 [15062.559534] LNet: Removed LNI 192.168.201.133@tcp115 [15063.583148] LNet: Removed LNI 192.168.201.133@tcp116 [15064.607157] LNet: Removed LNI 192.168.201.133@tcp117 [15065.631233] LNet: Removed LNI 192.168.201.133@tcp118 [15065.634538] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15065.640691] LNetError: 468197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15065.647143] LNet: Removed LNI 192.168.201.133@tcp119 [15066.655293] LNet: Removed LNI 192.168.201.133@tcp120 [15066.665334] LNet: Removed LNI 192.168.201.133@tcp121 [15067.679224] LNet: Removed LNI 192.168.201.133@tcp122 [15068.705468] LNet: Removed LNI 192.168.201.133@tcp123 [15069.727371] LNet: Removed LNI 192.168.201.133@tcp124 [15070.751151] LNet: Removed LNI 192.168.201.133@tcp125 [15071.775654] LNet: Removed LNI 192.168.201.133@tcp126 [15072.799202] LNet: Removed LNI 192.168.201.133@tcp127 [15073.825150] LNet: Removed LNI 192.168.201.133@tcp128 [15073.828978] LNet: Removed LNI 192.168.201.133@tcp129 [15074.847092] LNet: Removed LNI 192.168.201.133@tcp130 [15075.871101] LNet: Removed LNI 192.168.201.133@tcp131 [15076.895575] LNet: Removed LNI 192.168.201.133@tcp132 [15076.900238] LNet: Removed LNI 192.168.201.133@tcp133 [15076.903300] LNet: Removed LNI 192.168.201.133@tcp134 [15077.930674] LNet: Removed LNI 192.168.201.133@tcp135 [15078.683781] Key type .llcrypt unregistered [15078.685582] Key type ._llcrypt unregistered [15099.771671] Key type ._llcrypt registered [15099.773871] Key type .llcrypt registered [15099.897264] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [15112.677798] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [15113.752682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15113.796191] alg: No test for adler32 (adler32-zlib) [15114.972744] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [15115.216051] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [15116.919174] Key type lgssc registered [15118.021616] Lustre: Echo OBD driver; http://www.lustre.org/ [15151.428722] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:28:30 (1786764510) [15170.129413] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [15173.663463] Key type lgssc unregistered [15173.968567] LNet: 472270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15173.976494] LNetError: 472270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15173.989923] LNet: Removed LNI 192.168.201.133@tcp [15175.171618] Key type .llcrypt unregistered [15175.173918] Key type ._llcrypt unregistered [15192.247428] Key type ._llcrypt registered [15192.249357] Key type .llcrypt registered [15192.330086] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [15193.333917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15193.458231] alg: No test for adler32 (adler32-zlib) [15194.629970] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [15194.894499] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [15196.583281] Key type lgssc registered [15197.839325] Lustre: Echo OBD driver; http://www.lustre.org/ [15208.285869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15208.315048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15209.577299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15209.623347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15209.745406] Lustre: lustre-MDT0000: new disk, initializing [15209.901468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15209.934504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15214.794921] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15226.985807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15227.093405] Lustre: 473489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15227.158533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15227.166699] Lustre: Skipped 1 previous similar message [15227.248297] Lustre: lustre-MDT0001: new disk, initializing [15227.378123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15227.406374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15227.417744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15232.732979] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15243.552327] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15250.038349] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15256.582299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15256.841213] Lustre: lustre-OST0000: new disk, initializing [15256.845228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15256.849801] Lustre: 475458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15256.952929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15262.771270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15262.777024] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15262.890633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15263.263924] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15272.566786] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15277.148603] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing lnet_if_list [15283.171346] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15283.212288] LNet: Added LNI 192.168.201.133@tcp10 [8/256/0/180] [15288.473376] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15288.503328] LNet: Added LNI 192.168.201.133@tcp11 [8/256/0/180] [15292.766915] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15292.817681] LNet: Added LNI 192.168.201.133@tcp12 [8/256/0/180] [15298.107643] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15298.153860] LNet: Added LNI 192.168.201.133@tcp13 [8/256/0/180] [15303.287996] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15303.343646] LNet: Added LNI 192.168.201.133@tcp14 [8/256/0/180] [15309.094469] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15309.117586] LNet: Added LNI 192.168.201.133@tcp15 [8/256/0/180] [15313.998711] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15314.025319] LNet: Added LNI 192.168.201.133@tcp16 [8/256/0/180] [15320.115623] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15320.142867] LNet: Added LNI 192.168.201.133@tcp17 [8/256/0/180] [15326.667241] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15326.728480] LNet: Added LNI 192.168.201.133@tcp18 [8/256/0/180] [15333.444481] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15333.533773] LNet: Added LNI 192.168.201.133@tcp19 [8/256/0/180] [15340.993239] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15341.012050] LNet: Added LNI 192.168.201.133@tcp20 [8/256/0/180] [15346.713626] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15346.736398] LNet: Added LNI 192.168.201.133@tcp21 [8/256/0/180] [15352.332569] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15352.357074] LNet: Added LNI 192.168.201.133@tcp22 [8/256/0/180] [15360.092694] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15360.117464] LNet: Added LNI 192.168.201.133@tcp23 [8/256/0/180] [15366.945469] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15366.978320] LNet: Added LNI 192.168.201.133@tcp24 [8/256/0/180] [15373.980669] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15374.046611] LNet: Added LNI 192.168.201.133@tcp25 [8/256/0/180] [15381.740948] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15381.775544] LNet: Added LNI 192.168.201.133@tcp26 [8/256/0/180] [15387.895204] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15387.932076] LNet: Added LNI 192.168.201.133@tcp27 [8/256/0/180] [15394.877230] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15394.901619] LNet: Added LNI 192.168.201.133@tcp28 [8/256/0/180] [15401.248559] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15401.270690] LNet: Added LNI 192.168.201.133@tcp29 [8/256/0/180] [15407.436454] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15407.457387] LNet: Added LNI 192.168.201.133@tcp30 [8/256/0/180] [15414.291739] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15414.331503] LNet: Added LNI 192.168.201.133@tcp31 [8/256/0/180] [15420.020649] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15420.039716] LNet: Added LNI 192.168.201.133@tcp32 [8/256/0/180] [15426.636718] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15426.675928] LNet: Added LNI 192.168.201.133@tcp33 [8/256/0/180] [15433.005348] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15433.030031] LNet: Added LNI 192.168.201.133@tcp34 [8/256/0/180] [15439.711195] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15439.740091] LNet: Added LNI 192.168.201.133@tcp35 [8/256/0/180] [15445.982193] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15446.003962] LNet: Added LNI 192.168.201.133@tcp36 [8/256/0/180] [15453.530387] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15453.562971] LNet: Added LNI 192.168.201.133@tcp37 [8/256/0/180] [15461.098931] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15461.120680] LNet: Added LNI 192.168.201.133@tcp38 [8/256/0/180] [15470.593351] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15470.627740] LNet: Added LNI 192.168.201.133@tcp39 [8/256/0/180] [15479.315888] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15479.333479] LNet: Added LNI 192.168.201.133@tcp40 [8/256/0/180] [15486.533304] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15486.558841] LNet: Added LNI 192.168.201.133@tcp41 [8/256/0/180] [15495.267542] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15495.314188] LNet: Added LNI 192.168.201.133@tcp42 [8/256/0/180] [15505.240172] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15505.283508] LNet: Added LNI 192.168.201.133@tcp43 [8/256/0/180] [15514.041378] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15514.097383] LNet: Added LNI 192.168.201.133@tcp44 [8/256/0/180] [15522.581518] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15522.647549] LNet: Added LNI 192.168.201.133@tcp45 [8/256/0/180] [15532.186459] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15532.214634] LNet: Added LNI 192.168.201.133@tcp46 [8/256/0/180] [15541.015780] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15541.063822] LNet: Added LNI 192.168.201.133@tcp47 [8/256/0/180] [15549.150506] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15549.185504] LNet: Added LNI 192.168.201.133@tcp48 [8/256/0/180] [15557.183674] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15557.212313] LNet: Added LNI 192.168.201.133@tcp49 [8/256/0/180] [15566.467847] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15566.505673] LNet: Added LNI 192.168.201.133@tcp50 [8/256/0/180] [15585.766194] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15585.781673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15585.784536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15585.794032] Lustre: Skipped 2 previous similar messages [15590.882813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15590.930671] Lustre: Skipped 2 previous similar messages [15592.429569] Lustre: server umount lustre-MDT0000 complete [15597.064623] LustreError: 473480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786764957 with bad export cookie 399582159774478616 [15597.076849] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15597.083059] LustreError: 473480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15597.643867] Lustre: server umount lustre-MDT0001 complete [15617.503206] Lustre: 472761:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786764961/real 1786764961] req@00000000af681c52 x1873558430202624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786764977 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15617.527798] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15617.535551] Lustre: Skipped 2 previous similar messages [15618.527641] Lustre: 472760:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786764962/real 1786764962] req@0000000087c82e9a x1873558430202752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786764978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15619.738513] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [15622.623407] Lustre: 472758:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786764966/real 1786764966] req@000000001d007e83 x1873558430202880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786764982 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15627.746050] Lustre: 472761:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786764972/real 1786764972] req@000000006cb9d23e x1873558430203264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786764988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15627.766198] Lustre: 472761:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15629.379252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15632.863150] Lustre: 472760:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786764977/real 1786764977] req@000000007dbf95ad x1873558430203392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786764993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15642.568409] LustreError: 475452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15642.586310] LustreError: 475452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15642.636080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15647.719707] LustreError: 475453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15648.245370] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15650.798897] LustreError: 475452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15650.802123] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15655.905348] LustreError: 475453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15656.262538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15656.741108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15661.885171] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15662.401176] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15671.744963] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15678.568914] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15703.012445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15703.033636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15703.037389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15703.056826] Lustre: Skipped 1 previous similar message [15706.815995] Lustre: server umount lustre-OST0000 complete [15713.248314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15713.250844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15713.266101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15713.280728] Lustre: Skipped 2 previous similar messages [15713.289953] Lustre: Skipped 1 previous similar message [15717.622432] Lustre: server umount lustre-MDT0000 complete [15718.368898] LustreError: 499706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15718.387079] LustreError: 499706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15721.649113] LustreError: 474285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786765081 with bad export cookie 399582159774481367 [15721.649858] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15721.658915] LustreError: 474285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15721.978240] Lustre: server umount lustre-MDT0001 complete [15734.779354] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [15738.099899] Key type lgssc unregistered [15738.509077] LNet: 502514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15738.520953] LNetError: 502514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15738.527620] LNet: Removed LNI 192.168.201.133@tcp [15738.530910] LNet: Removed LNI 192.168.201.133@tcp10 [15738.533879] LNet: Removed LNI 192.168.201.133@tcp11 [15738.537226] LNet: Removed LNI 192.168.201.133@tcp12 [15738.541264] LNet: Removed LNI 192.168.201.133@tcp13 [15738.543860] LNet: Removed LNI 192.168.201.133@tcp14 [15738.547237] LNet: Removed LNI 192.168.201.133@tcp15 [15738.551394] LNet: Removed LNI 192.168.201.133@tcp16 [15738.554456] LNet: Removed LNI 192.168.201.133@tcp17 [15738.557941] LNet: Removed LNI 192.168.201.133@tcp18 [15738.561610] LNet: Removed LNI 192.168.201.133@tcp19 [15738.564951] LNet: Removed LNI 192.168.201.133@tcp20 [15738.567862] LNet: Removed LNI 192.168.201.133@tcp21 [15738.570600] LNet: Removed LNI 192.168.201.133@tcp22 [15738.573543] LNet: Removed LNI 192.168.201.133@tcp23 [15738.576509] LNet: Removed LNI 192.168.201.133@tcp24 [15738.579783] LNet: Removed LNI 192.168.201.133@tcp25 [15738.582620] LNet: Removed LNI 192.168.201.133@tcp26 [15738.584625] LNet: Removed LNI 192.168.201.133@tcp27 [15738.587467] LNet: Removed LNI 192.168.201.133@tcp28 [15738.590336] LNet: Removed LNI 192.168.201.133@tcp29 [15738.592693] LNet: Removed LNI 192.168.201.133@tcp30 [15738.595586] LNet: Removed LNI 192.168.201.133@tcp31 [15738.598539] LNet: Removed LNI 192.168.201.133@tcp32 [15738.601599] LNet: Removed LNI 192.168.201.133@tcp33 [15738.604503] LNet: Removed LNI 192.168.201.133@tcp34 [15738.607845] LNet: Removed LNI 192.168.201.133@tcp35 [15738.610955] LNet: Removed LNI 192.168.201.133@tcp36 [15738.614609] LNet: Removed LNI 192.168.201.133@tcp37 [15738.616756] LNet: Removed LNI 192.168.201.133@tcp38 [15738.619927] LNet: Removed LNI 192.168.201.133@tcp39 [15738.624678] LNet: Removed LNI 192.168.201.133@tcp40 [15738.627185] LNet: Removed LNI 192.168.201.133@tcp41 [15738.631556] LNet: Removed LNI 192.168.201.133@tcp42 [15738.635624] LNet: Removed LNI 192.168.201.133@tcp43 [15738.638265] LNet: Removed LNI 192.168.201.133@tcp44 [15738.641710] LNet: Removed LNI 192.168.201.133@tcp45 [15738.644323] LNet: Removed LNI 192.168.201.133@tcp46 [15738.648879] LNet: Removed LNI 192.168.201.133@tcp47 [15738.652255] LNet: Removed LNI 192.168.201.133@tcp48 [15738.657536] LNet: Removed LNI 192.168.201.133@tcp49 [15738.668552] LNet: Removed LNI 192.168.201.133@tcp50 [15739.554151] Key type .llcrypt unregistered [15739.556072] Key type ._llcrypt unregistered [15761.543759] Key type ._llcrypt registered [15761.545862] Key type .llcrypt registered [15761.605454] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [15777.653627] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [15778.774371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15778.939067] alg: No test for adler32 (adler32-zlib) [15780.226354] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [15780.489260] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [15782.143221] Key type lgssc registered [15783.248074] Lustre: Echo OBD driver; http://www.lustre.org/ [15819.703414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15819.722228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15820.960977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15820.986688] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15821.064681] Lustre: lustre-MDT0000: new disk, initializing [15821.135110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15821.151265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15826.037271] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15836.763447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15836.910272] Lustre: 505713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15836.966973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15836.970341] Lustre: Skipped 1 previous similar message [15837.032418] Lustre: lustre-MDT0001: new disk, initializing [15837.087487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15837.129649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15837.142449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15841.034909] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15851.159360] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15858.205351] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15868.830695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15869.214321] Lustre: *** cfs_fail_loc=90f, val=0*** [15869.331079] Lustre: lustre-OST0000: new disk, initializing [15869.335790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15869.342355] Lustre: 507728:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15869.515733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15874.830842] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15877.166485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15877.178595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15877.332846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15885.466391] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15898.083143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15898.110215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15898.158209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15898.169703] Lustre: Skipped 1 previous similar message [15899.840316] Lustre: server umount lustre-OST0000 complete [15903.713390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15903.718561] LustreError: Skipped 1 previous similar message [15903.727017] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15903.739588] Lustre: Skipped 1 previous similar message [15903.746122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15908.327445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15908.328160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15908.353717] Lustre: Skipped 1 previous similar message [15910.157235] Lustre: server umount lustre-MDT0000 complete [15914.103635] LustreError: 505703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786765274 with bad export cookie 14420882524099404262 [15914.117644] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15914.124343] LustreError: 505703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15914.499397] Lustre: server umount lustre-MDT0001 complete [15926.012283] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [15928.829387] Key type lgssc unregistered [15929.126776] LNet: 509637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15929.131899] LNetError: 509637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15929.148904] LNet: Removed LNI 192.168.201.133@tcp [15930.062322] Key type .llcrypt unregistered [15930.065460] Key type ._llcrypt unregistered [15948.924393] Key type ._llcrypt registered [15948.926978] Key type .llcrypt registered [15949.074169] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [15950.482991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15950.518343] alg: No test for adler32 (adler32-zlib) [15951.598057] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [15951.818100] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [15953.491906] Key type lgssc registered [15954.701167] Lustre: Echo OBD driver; http://www.lustre.org/ [15962.731773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15962.743196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15964.146568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15968.533726] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15974.376355] LustreError: 510921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15974.416844] LustreError: 510921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15977.601575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15978.281092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15984.739170] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15993.024973] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15999.429995] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16007.120503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16007.353309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16013.152955] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16020.753175] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16035.036305] loop0: detected capacity change from 0 to 204800000 [16035.164926] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16035.191799] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16035.208708] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16041.899124] loop0: detected capacity change from 0 to 204800000 [16042.136987] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16042.146157] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16042.153516] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16047.278804] loop0: detected capacity change from 0 to 204800000 [16053.747376] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16053.838509] Lustre: fs15246-OST0000: new disk, initializing [16053.842387] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16053.846958] Lustre: 514521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16053.914356] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16060.745225] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16071.542789] loop1: detected capacity change from 0 to 204800000 [16078.572502] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16078.686924] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16078.750364] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16078.883192] Lustre: fs15246-MDT0000: new disk, initializing [16079.137981] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16079.173597] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16079.349431] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16080.417070] Lustre: 515644:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b0ddeae4 x1873559223847808/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786765456 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16082.929992] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16085.493314] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16085.504456] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16085.553327] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16110.579458] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16110.597822] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16112.724201] Lustre: server umount fs15246-MDT0000 complete [16115.690417] LustreError: 512934:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16120.804701] LustreError: 510918:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16123.731461] Lustre: server umount fs15246-OST0000 complete [16136.162175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16136.177544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16136.185951] Lustre: Skipped 1 previous similar message [16136.209173] Lustre: Skipped 1 previous similar message [16141.314103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16142.278570] Lustre: server umount lustre-OST0000 complete [16146.402059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16146.412339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16146.416331] Lustre: Skipped 1 previous similar message [16146.436106] Lustre: Skipped 2 previous similar messages [16151.521503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16151.530587] Lustre: Skipped 1 previous similar message [16151.937925] Lustre: server umount lustre-MDT0000 complete [16155.717235] LustreError: 510902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786765515 with bad export cookie 13180135687170903369 [16155.725807] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16155.732806] LustreError: 510902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16155.987628] Lustre: server umount lustre-MDT0001 complete [16166.333617] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [16168.851466] Key type lgssc unregistered [16169.172695] LNet: 518744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16169.178841] LNetError: 518744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16169.188876] LNet: Removed LNI 192.168.201.133@tcp [16169.933185] Key type .llcrypt unregistered [16169.935594] Key type ._llcrypt unregistered [16189.967648] Key type ._llcrypt registered [16189.970376] Key type .llcrypt registered [16190.122317] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [16192.646694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16192.677186] alg: No test for adler32 (adler32-zlib) [16193.861256] Lustre: Lustre: Build Version: 2.17.57_2_g67a826c [16194.279904] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [16196.127193] Key type lgssc registered [16197.735909] Lustre: Echo OBD driver; http://www.lustre.org/ [16247.594202] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [16258.753957] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [16294.952232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16295.004197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16296.394627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16296.446064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16296.541370] Lustre: lustre-MDT0000: new disk, initializing [16296.725185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16296.750073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16301.583688] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16312.360916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16312.487552] Lustre: 523211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16312.557859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16312.563086] Lustre: Skipped 1 previous similar message [16312.647504] Lustre: lustre-MDT0001: new disk, initializing [16312.782632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16312.814854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16312.826318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16317.500841] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16327.416679] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16334.887137] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16342.523247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16342.782485] Lustre: lustre-OST0000: new disk, initializing [16342.786455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16342.791709] Lustre: 525188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16342.877732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16349.505264] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16351.354590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16351.375597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16351.529541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16359.673455] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16364.741452] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16364.914237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16369.889252] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16370.176292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16377.318385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16377.337224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16377.343310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16377.374731] Lustre: Skipped 1 previous similar message [16378.481366] Lustre: server umount lustre-OST0000 complete [16384.483209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16384.493196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16384.512080] Lustre: Skipped 1 previous similar message [16384.521683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16387.553113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16387.554037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16387.576061] Lustre: Skipped 1 previous similar message [16388.796142] Lustre: server umount lustre-MDT0000 complete [16392.840175] LustreError: 523204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786765753 with bad export cookie 3951888829399948565 [16392.852219] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16392.857042] LustreError: 523204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16393.168366] Lustre: server umount lustre-MDT0001 complete [16418.944398] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [16427.950527] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [16460.893874] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16144 sec ========= 23:50:19 (1786765819) [16462.325896] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:50:21 (1786765821) === [16465.876152] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:50:24 (1786765824) === [16509.315786] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [16514.441309] Key type lgssc unregistered [16514.886068] LNet: 531836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16514.894516] LNetError: 531836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16514.926904] LNet: Removed LNI 192.168.201.133@tcp [16516.302067] Key type .llcrypt unregistered [16516.303504] Key type ._llcrypt unregistered