[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1376378278 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003162] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005025] kvm-guest: setup PV IPIs [ 0.009324] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011009] pid_max: default: 32768 minimum: 301 [ 0.013095] LSM: Security Framework initializing [ 0.014046] Yama: becoming mindful. [ 0.015031] SELinux: Initializing. [ 0.016088] *** VALIDATE selinux *** [ 0.027199] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.037607] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.038232] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.040114] *** VALIDATE tmpfs *** [ 0.042504] *** VALIDATE proc *** [ 0.044276] *** VALIDATE cgroup *** [ 0.045015] *** VALIDATE cgroup2 *** [ 0.046281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.047155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.048010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.049044] Spectre V2 : User space: Vulnerable [ 0.050017] Speculative Store Bypass: Vulnerable [ 0.055143] debug: unmapping init [mem 0xffffffff98859000-0xffffffff98860fff] [ 0.058608] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.061191] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.062024] ... version: 2 [ 0.063020] ... bit width: 48 [ 0.064037] ... generic registers: 4 [ 0.065099] ... value mask: 0000ffffffffffff [ 0.066018] ... max period: 00007fffffffffff [ 0.067019] ... fixed-purpose events: 3 [ 0.068020] ... event mask: 000000070000000f [ 0.070112] rcu: Hierarchical SRCU implementation. [ 0.073563] smp: Bringing up secondary CPUs ... [ 0.074563] x86: Booting SMP configuration: [ 0.076017] .... node #0, CPUs: #1 #2 #3 [ 0.233197] smp: Brought up 1 node, 4 CPUs [ 0.235017] smpboot: Max logical packages: 1 [ 0.236019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.379018] node 0 deferred pages initialised in 140ms [ 0.405332] devtmpfs: initialized [ 0.407028] x86/mm: Memory block size: 128MB [ 0.416987] gcov: version magic: 0x41383552 [ 0.423761] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.431194] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.438412] pinctrl core: initialized pinctrl subsystem [ 0.443933] [ 0.446010] ************************************************************* [ 0.451166] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.458053] ** ** [ 0.464113] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.469012] ** ** [ 0.473018] ** This means that this kernel is built to expose internal ** [ 0.476013] ** IOMMU data structures, which may compromise security on ** [ 0.480013] ** your system. ** [ 0.483013] ** ** [ 0.489037] ** If you see this message and you are not debugging the ** [ 0.494024] ** kernel, report this immediately to your vendor! ** [ 0.500014] ** ** [ 0.505014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.509014] ************************************************************* [ 0.516090] NET: Registered protocol family 16 [ 0.518736] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.522086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.527292] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.535419] cpuidle: using governor menu [ 0.540780] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.544757] PCI: Using configuration type 1 for base access [ 0.549136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.574131] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.581110] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.583147] cryptd: max_cpu_qlen set to 1000 [ 0.592000] ACPI: Added _OSI(Module Device) [ 0.595015] ACPI: Added _OSI(Processor Device) [ 0.597009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.600021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.608134] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.616477] ACPI: Interpreter enabled [ 0.617115] ACPI: PM: (supports S0 S3 S4 S5) [ 0.619014] ACPI: Using IOAPIC for interrupt routing [ 0.623180] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.630811] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.648323] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.651033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.654030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.659078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.667912] acpiphp: Slot [2] registered [ 0.669109] acpiphp: Slot [5] registered [ 0.672138] acpiphp: Slot [6] registered [ 0.673095] acpiphp: Slot [7] registered [ 0.674000] acpiphp: Slot [8] registered [ 0.674000] acpiphp: Slot [9] registered [ 0.674115] acpiphp: Slot [10] registered [ 0.676111] acpiphp: Slot [3] registered [ 0.677086] acpiphp: Slot [4] registered [ 0.679103] acpiphp: Slot [11] registered [ 0.680089] acpiphp: Slot [12] registered [ 0.682106] acpiphp: Slot [13] registered [ 0.684090] acpiphp: Slot [14] registered [ 0.685083] acpiphp: Slot [15] registered [ 0.687104] acpiphp: Slot [16] registered [ 0.689121] acpiphp: Slot [17] registered [ 0.690100] acpiphp: Slot [18] registered [ 0.692082] acpiphp: Slot [19] registered [ 0.694096] acpiphp: Slot [20] registered [ 0.696086] acpiphp: Slot [21] registered [ 0.697257] acpiphp: Slot [22] registered [ 0.698224] acpiphp: Slot [23] registered [ 0.700251] acpiphp: Slot [24] registered [ 0.702289] acpiphp: Slot [25] registered [ 0.704108] acpiphp: Slot [26] registered [ 0.706081] acpiphp: Slot [27] registered [ 0.708098] acpiphp: Slot [28] registered [ 0.709106] acpiphp: Slot [29] registered [ 0.711094] acpiphp: Slot [30] registered [ 0.712191] acpiphp: Slot [31] registered [ 0.713000] PCI host bridge to bus 0000:00 [ 0.713000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.713000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.713000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.713000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.713000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.713000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.713000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.713000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.713000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.713000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.714061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.715000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.715000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.722016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.724748] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.728115] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.735144] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.739021] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.744014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.761015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.769012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.774965] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.785015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.796014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.827055] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.851170] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.866034] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.883017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.902000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.912549] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.920013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.926017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.943020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.955521] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.965019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.977015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.992017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.002194] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.011023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.018019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.037016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.048300] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.055017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.063015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.067000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.067000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.067000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.067000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.067000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.067000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.068067] iommu: Default domain type: Passthrough [ 1.070433] SCSI subsystem initialized [ 1.072117] ACPI: bus type USB registered [ 1.073082] usbcore: registered new interface driver usbfs [ 1.075192] usbcore: registered new interface driver hub [ 1.077068] usbcore: registered new device driver usb [ 1.079171] pps_core: LinuxPPS API ver. 1 registered [ 1.081009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.084052] PTP clock support registered [ 1.086417] EDAC MC: Ver: 3.0.0 [ 1.087373] PCI: Using ACPI for IRQ routing [ 1.090086] NetLabel: Initializing [ 1.091000] NetLabel: domain hash size = 128 [ 1.091000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.094253] NetLabel: unlabeled traffic allowed by default [ 1.097465] vgaarb: loaded [ 1.099681] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.102008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.110684] clocksource: Switched to clocksource kvm-clock [ 1.725810] VFS: Disk quotas dquot_6.6.0 [ 1.733209] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.740327] *** VALIDATE ramfs *** [ 1.744406] *** VALIDATE hugetlbfs *** [ 1.749510] pnp: PnP ACPI init [ 1.755591] pnp: PnP ACPI: found 6 devices [ 1.795280] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.799236] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.801669] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.804639] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.808383] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.811146] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.816098] NET: Registered protocol family 2 [ 1.820266] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.827706] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.833440] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.841361] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.845580] TCP: Hash tables configured (established 65536 bind 65536) [ 1.849580] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.853693] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.858595] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.863370] NET: Registered protocol family 1 [ 1.868630] RPC: Registered named UNIX socket transport module. [ 1.872408] RPC: Registered udp transport module. [ 1.874658] RPC: Registered tcp transport module. [ 1.877335] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.885501] NET: Registered protocol family 44 [ 1.892401] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.899441] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.904553] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.911225] PCI: CLS 0 bytes, default 64 [ 1.917662] Unpacking initramfs... [ 4.561268] debug: unmapping init [mem 0xffff9884fcc54000-0xffff9884fffbffff] [ 4.565253] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.567525] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.570658] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.820856] Initialise system trusted keyrings [ 5.825556] Key type blacklist registered [ 5.828344] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.838955] zbud: loaded [ 5.842346] *** VALIDATE nfs *** [ 5.843784] *** VALIDATE nfs4 *** [ 5.845405] pstore: using deflate compression [ 5.850543] Platform Keyring initialized [ 6.148967] NET: Registered protocol family 38 [ 6.152868] Key type asymmetric registered [ 6.159406] Asymmetric key parser 'x509' registered [ 6.163218] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.172322] io scheduler mq-deadline registered [ 6.173744] io scheduler kyber registered [ 6.175183] io scheduler bfq registered [ 6.176727] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.179389] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.181942] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.184526] ACPI: Power Button [PWRF] [ 6.189506] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.196636] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.210101] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.217292] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.237655] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.267488] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.297932] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.303242] Non-volatile memory driver v1.3 [ 6.305103] Linux agpgart interface v0.103 [ 6.338336] virtio_blk virtio1: [vda] 134496 512-byte logical blocks (68.9 MB/65.7 MiB) [ 6.341367] vda: detected capacity change from 0 to 68861952 [ 6.361324] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.364549] vdb: detected capacity change from 0 to 1073741824 [ 6.502365] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.509573] vdc: detected capacity change from 0 to 2621440000 [ 6.526390] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.529210] vdd: detected capacity change from 0 to 2621440000 [ 6.544835] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.548157] vde: detected capacity change from 0 to 4294967296 [ 6.572942] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.585364] vdf: detected capacity change from 0 to 4294967296 [ 6.593938] libphy: Fixed MDIO Bus: probed [ 6.600782] usbcore: registered new interface driver usbserial_generic [ 6.603639] usbserial: USB Serial support registered for generic [ 6.606400] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.611477] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.613475] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.616379] mousedev: PS/2 mouse device common for all mice [ 6.619848] rtc_cmos 00:05: RTC can wake from S4 [ 6.625039] rtc_cmos 00:05: registered as rtc0 [ 6.626780] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.629485] intel_pstate: CPU model not supported [ 6.632526] hid: raw HID events driver (C) Jiri Kosina [ 6.635193] usbcore: registered new interface driver usbhid [ 6.647706] usbhid: USB HID core driver [ 6.654355] drop_monitor: Initializing network drop monitor service [ 6.658141] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.669442] Initializing XFRM netlink socket [ 6.671492] NET: Registered protocol family 10 [ 6.676755] Segment Routing with IPv6 [ 6.679492] NET: Registered protocol family 17 [ 6.681078] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.690222] mpls_gso: MPLS GSO support [ 6.698550] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.707159] RAS: Correctable Errors collector initialized. [ 6.710108] AVX version of gcm_enc/dec engaged. [ 6.711945] AES CTR mode by8 optimization enabled [ 6.839439] sched_clock: Marking stable (6839417727, 0)->(10197156957, -3357739230) [ 6.845782] registered taskstats version 1 [ 6.848993] Loading compiled-in X.509 certificates [ 6.853226] zswap: loaded using pool lzo/zbud [ 6.883844] Key type big_key registered [ 6.899922] Key type encrypted registered [ 6.902288] ima: No TPM chip found, activating TPM-bypass! [ 6.904991] ima: Allocated hash algorithm: sha1 [ 6.907319] ima: No architecture policies found [ 6.909563] evm: Initialising EVM extended attributes: [ 6.912420] evm: security.selinux [ 6.913990] evm: security.ima [ 6.915497] evm: security.capability [ 6.917279] evm: HMAC attrs: 0x1 [ 6.920494] rtc_cmos 00:05: setting system clock to 2026-04-29 00:40:23 UTC (1777423223) [ 6.928106] debug: unmapping init [mem 0xffffffff99803000-0xffffffff999fffff] [ 6.931936] debug: unmapping init [mem 0xffffffff98582000-0xffffffff98858fff] [ 6.940193] Write protecting the kernel read-only data: 28672k [ 6.943616] debug: unmapping init [mem 0xffffffff96c03000-0xffffffff96dfffff] [ 6.946570] debug: unmapping init [mem 0xffffffff97514000-0xffffffff975fffff] [ 6.998844] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.008957] systemd[1]: Detected virtualization kvm. [ 7.011378] systemd[1]: Detected architecture x86-64. [ 7.013612] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.045146] systemd[1]: No hostname configured. [ 7.046760] systemd[1]: Set hostname to . [ 7.049225] random: systemd: uninitialized urandom read (16 bytes read) [ 7.052320] systemd[1]: Initializing machine ID from random generator. [ 7.182308] random: ln: uninitialized urandom read (6 bytes read) [ 7.427392] random: systemd: uninitialized urandom read (16 bytes read) [ 7.430284] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 7.440407] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 7.444644] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.960304] device-mapper: uevent: version 1.0.3 [ 8.962639] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices...[ 9.551550] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ 10.190805] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.553979] scsi host0: ata_piix [ 10.653646] scsi host1: ata_piix [ 10.658362] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.663330] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.440467] random: crng init done [ 16.445064] random: 7 urandom warning(s) missed due to ratelimiting [ 18.444115] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.558488] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.419083] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.830476] SELinux: Disabled at runtime. [ 26.023921] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.060249] systemd[1]: Detected virtualization kvm. [ 26.073889] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.805617] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.820226] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.834912] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.846429] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.854861] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.880313] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.895155] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ 29.169629] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 30.914694] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.759270] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.779284] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.386136] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.520226] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 40.203078] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 41.426062] NFS: Registering the id_resolver key type [ 41.432574] Key type id_resolver registered [ 41.434140] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg234-server login: [ 86.823154] hrtimer: interrupt took 8699771 ns [ 98.820168] libcfs: loading out-of-tree module taints kernel. [ 98.858383] Key type ._llcrypt registered [ 98.861685] Key type .llcrypt registered [ 98.931283] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 110.510206] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 111.606907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.616097] alg: No test for adler32 (adler32-zlib) [ 112.860392] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 113.533683] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 115.231716] Key type lgssc registered [ 116.531929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.001710] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 157.878329] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 167.276370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 167.294968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 168.504445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 168.521237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 168.597585] Lustre: lustre-MDT0000: new disk, initializing [ 168.684517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.706455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.797964] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.372665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.456654] Lustre: 6476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 181.481342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 181.484744] Lustre: Skipped 1 previous similar message [ 181.538630] Lustre: lustre-MDT0001: new disk, initializing [ 181.571423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.592983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 181.604566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.577646] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.539819] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 194.692498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.874239] Lustre: lustre-OST0000: new disk, initializing [ 194.877583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 194.925602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.954200] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.701571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.714146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.757437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 207.702196] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.765303] Lustre: lustre-OST0001: new disk, initializing [ 207.766952] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.793583] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.757744] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.533621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.542648] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.586243] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 222.653298] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.040739] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.689371] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing check_logdir /tmp/testlogs/ [ 239.638453] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing yml_node [ 243.169291] Lustre: DEBUG MARKER: Client: 2.17.51.75 [ 245.384480] Lustre: DEBUG MARKER: MDS: 2.17.51.75 [ 247.230693] Lustre: DEBUG MARKER: OSS: 2.17.51.75 [ 248.332213] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 28 20:44:23 EDT 2026 [ 260.735902] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 261.812604] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 279.008784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 279.012476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.036731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.033721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.038462] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.049792] Lustre: Skipped 1 previous similar message [ 285.153712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 285.160042] Lustre: Skipped 5 previous similar messages [ 290.272572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.273047] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 290.281556] Lustre: Skipped 3 previous similar messages [ 290.485578] Lustre: server umount lustre-MDT0000 complete [ 293.583041] LustreError: 6468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423510 with bad export cookie 7427607548189288209 [ 293.584648] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.600313] LustreError: 6468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 295.392479] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.404495] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 295.411696] Lustre: Skipped 2 previous similar messages [ 299.829866] Lustre: server umount lustre-MDT0001 complete [ 303.141891] Lustre: server umount lustre-OST0000 complete [ 306.639796] Lustre: server umount lustre-OST0001 complete [ 311.705582] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 318.732767] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 346.678235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.882931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 346.902499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 346.954929] Lustre: lustre-MDT0000: new disk, initializing [ 347.038102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.068075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 350.493150] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 358.882934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.978861] Lustre: 14948:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 359.040608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 359.044612] Lustre: Skipped 1 previous similar message [ 359.160372] Lustre: lustre-MDT0001: new disk, initializing [ 359.272746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 359.338415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 359.352362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 363.176176] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 369.532551] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 372.415735] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 377.969580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 378.276124] Lustre: lustre-OST0000: new disk, initializing [ 378.282143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 378.360490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 379.667191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 379.677969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 379.814242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 383.289984] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 389.478960] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 392.529789] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 392.680627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 395.781209] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 395.943255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 400.358296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 400.359273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.365236] LustreError: Skipped 1 previous similar message [ 400.366483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 400.366489] Lustre: Skipped 3 previous similar messages [ 400.387020] Lustre: Skipped 1 previous similar message [ 403.654642] Lustre: server umount lustre-OST0000 complete [ 410.593726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.609768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 420.834480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 420.844362] Lustre: Skipped 6 previous similar messages [ 421.346894] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 421.571402] Lustre: server umount lustre-MDT0000 complete [ 424.649864] LustreError: 14941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423641 with bad export cookie 7427607548189289728 [ 424.661221] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 424.661840] LustreError: 14941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 425.011990] Lustre: server umount lustre-MDT0001 complete [ 428.354402] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:47:23 (1777423643) [ 430.713824] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 432.405616] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:47:27 (1777423647) [ 438.542541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.956560] LustreError: 18952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.975921] LustreError: 18952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 439.016530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.482540] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 444.385196] LustreError: 18953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.402508] LustreError: 18953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 448.485497] LustreError: 18952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 449.660963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.436875] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 458.270654] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 461.261187] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 466.583463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.599347] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 476.105626] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 494.560296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.564384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.572100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.580070] Lustre: Skipped 2 previous similar messages [ 494.600698] Lustre: Skipped 1 previous similar message [ 498.063496] Lustre: server umount lustre-OST0000 complete [ 501.223434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 507.127932] Lustre: server umount lustre-MDT0000 complete [ 509.926359] LustreError: 20005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.945368] LustreError: 20005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 510.074780] LustreError: 18934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423726 with bad export cookie 7427607548189290610 [ 510.084912] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.087158] LustreError: 18934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 510.290612] Lustre: server umount lustre-MDT0001 complete [ 518.474430] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 521.015200] Key type lgssc unregistered [ 521.312599] LNet: 22152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.319776] LNetError: 22152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.328670] LNet: Removed LNI 192.168.202.134@tcp [ 521.939151] Key type .llcrypt unregistered [ 521.942687] Key type ._llcrypt unregistered [ 541.329221] Key type ._llcrypt registered [ 541.331382] Key type .llcrypt registered [ 541.424979] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 542.089355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 542.155014] alg: No test for adler32 (adler32-zlib) [ 543.127688] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 543.282327] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 544.951345] Key type lgssc registered [ 545.829966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 551.988675] loop: module loaded [ 552.029187] loop0: detected capacity change from 0 to 204800000 [ 552.124416] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 552.144123] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 552.341848] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 556.934690] loop0: detected capacity change from 0 to 204800000 [ 557.028814] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 557.049817] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 557.072636] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 561.042382] loop0: detected capacity change from 0 to 204800000 [ 564.827116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 564.855796] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.101149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 566.161056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 566.223393] Lustre: lustre-MDT0000: new disk, initializing [ 566.300452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.318562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 569.840470] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 576.739368] loop1: detected capacity change from 0 to 204800000 [ 581.391208] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 581.634422] Lustre: lustre-OST07c6: new disk, initializing [ 581.646227] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 581.651490] Lustre: Skipped 1 previous similar message [ 581.713313] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 582.040531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 582.045719] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 582.088935] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 587.031174] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 592.245412] Lustre: 24303:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 592.256535] Lustre: 24303:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 592.261343] Lustre: 24303:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 592.266683] Lustre: 24303:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 592.271921] Lustre: 24303:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 592.276583] Lustre: 24303:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 613.285597] Lustre: 24301:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 613.292423] Lustre: 24301:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 613.297420] Lustre: 24301:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 613.307896] Lustre: 24301:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 613.315731] Lustre: 24301:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 613.323541] Lustre: 24301:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 617.819801] Lustre: 24301:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 617.828935] Lustre: 24301:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 617.836871] Lustre: 24301:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 617.844444] Lustre: 24301:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 617.850319] Lustre: 24301:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 617.857213] Lustre: 24301:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 617.860939] Lustre: 24301:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 617.867849] Lustre: 24301:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 617.873750] Lustre: 24301:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 617.878086] Lustre: 24301:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 617.882795] Lustre: 24301:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 617.887911] Lustre: 24301:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 621.054717] Lustre: 24303:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 621.064276] Lustre: 24303:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 621.069115] Lustre: 24303:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 621.075127] Lustre: 24303:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 621.080239] Lustre: 24303:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 621.085297] Lustre: 24303:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 621.090750] Lustre: 24303:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 621.095872] Lustre: 24303:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 621.101202] Lustre: 24303:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 621.107360] Lustre: 24303:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 621.115845] Lustre: 24303:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 621.125194] Lustre: 24303:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 625.368889] Lustre: 24301:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 625.376909] Lustre: 24301:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 625.382050] Lustre: 24301:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 625.387532] Lustre: 24301:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 625.396491] Lustre: 24301:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 625.402859] Lustre: 24301:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 625.411572] Lustre: 24301:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 625.416157] Lustre: 24301:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 625.422294] Lustre: 24301:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 625.426518] Lustre: 24301:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 625.432305] Lustre: 24301:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 625.436732] Lustre: 24301:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 635.984978] Lustre: 24302:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 635.992296] Lustre: 24302:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 5 previous similar messages [ 635.998344] Lustre: 24302:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 636.004070] Lustre: 24302:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 636.009683] Lustre: 24302:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 636.014356] Lustre: 24302:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 636.018891] Lustre: 24302:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 636.023787] Lustre: 24302:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 636.029290] Lustre: 24302:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 636.033763] Lustre: 24302:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 636.037892] Lustre: 24302:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 636.042921] Lustre: 24302:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 684.513647] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.523078] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 687.746481] Lustre: server umount lustre-OST07c6 complete [ 693.891966] Lustre: server umount lustre-MDT0000 complete [ 703.496963] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 705.067202] Key type lgssc unregistered [ 705.269701] LNet: 27819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.274313] LNetError: 27819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.287836] LNet: Removed LNI 192.168.202.134@tcp [ 705.739408] Key type .llcrypt unregistered [ 705.740902] Key type ._llcrypt unregistered [ 716.209585] Key type ._llcrypt registered [ 716.212478] Key type .llcrypt registered [ 716.261386] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 716.739825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 716.807346] alg: No test for adler32 (adler32-zlib) [ 717.757808] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 717.888368] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 719.479434] Key type lgssc registered [ 720.054155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 724.140247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 724.158112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.444971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 727.451387] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 733.301588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 733.511234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 736.451714] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 740.315339] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 742.656411] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 746.990486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 747.245706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.343131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 751.424091] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 755.185617] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 770.022936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 770.027033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.033515] LustreError: Skipped 1 previous similar message [ 770.049552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 771.868986] Lustre: server umount lustre-OST0000 complete [ 774.628105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 774.643580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.656502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.232481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.233915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 780.561886] Lustre: server umount lustre-MDT0000 complete [ 782.875285] LustreError: 29031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423999 with bad export cookie 1407657766143924693 [ 782.883728] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 793.153954] Lustre: server umount lustre-MDT0001 complete [ 799.444018] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 801.211031] Key type lgssc unregistered [ 801.408213] LNet: 32430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.413615] LNetError: 32430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.421555] LNet: Removed LNI 192.168.202.134@tcp [ 801.846212] Key type .llcrypt unregistered [ 801.848059] Key type ._llcrypt unregistered [ 813.674735] Key type ._llcrypt registered [ 813.676968] Key type .llcrypt registered [ 813.749179] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 814.458985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.475821] alg: No test for adler32 (adler32-zlib) [ 815.461514] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 815.639771] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 817.280863] Key type lgssc registered [ 818.110287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 823.081448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 823.103991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.428191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 826.965181] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 832.399322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.625474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 834.976416] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 838.495650] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 840.893105] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 845.242504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 845.510957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.626188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 848.960798] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 852.568525] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 864.228691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 864.233436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.239337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 868.321236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 868.324674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 868.327545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.347311] Lustre: Skipped 1 previous similar message [ 870.268280] Lustre: server umount lustre-OST0000 complete [ 873.441567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 873.443915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.452468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.464247] Lustre: Skipped 1 previous similar message [ 873.474749] Lustre: Skipped 1 previous similar message [ 878.560924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.813532] Lustre: server umount lustre-MDT0000 complete [ 881.167386] LustreError: 35925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424097 with bad export cookie 10440969818740654807 [ 881.173926] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 881.182220] LustreError: 35925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 881.352606] Lustre: server umount lustre-MDT0001 complete [ 887.630022] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 889.615520] Key type lgssc unregistered [ 889.783672] LNet: 36859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.788352] LNetError: 36859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.855925] LNet: Removed LNI 192.168.202.134@tcp [ 891.390794] Key type .llcrypt unregistered [ 891.395078] Key type ._llcrypt unregistered [ 906.303018] Key type ._llcrypt registered [ 906.306056] Key type .llcrypt registered [ 906.393090] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 907.117958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.128821] alg: No test for adler32 (adler32-zlib) [ 908.041602] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 908.238667] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 909.855202] Key type lgssc registered [ 910.651358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 916.181984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 916.189997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.469588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.077701] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 925.070195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.300115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 927.577389] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 930.902952] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 933.047625] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 936.997566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 937.207167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.119671] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 943.114294] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 943.349227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 954.851587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.854027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.860145] Lustre: Skipped 2 previous similar messages [ 954.871308] Lustre: Skipped 2 previous similar messages [ 957.252865] Lustre: server umount lustre-MDT0000 complete [ 959.594889] LustreError: 38218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424176 with bad export cookie 7595022479064575718 [ 959.599951] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 959.607466] LustreError: 38218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 959.967989] LustreError: 38233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.968347] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.977629] LustreError: 38233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 959.983140] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 959.989650] Lustre: Skipped 1 previous similar message [ 965.092926] LustreError: 38263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.099538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 965.113484] LustreError: 38263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 965.120112] Lustre: Skipped 1 previous similar message [ 965.896997] Lustre: server umount lustre-MDT0001 complete [ 993.249203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 993.378362] Lustre: server umount lustre-OST0000 complete [ 1002.558816] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1004.205582] Key type lgssc unregistered [ 1004.392331] LNet: 41744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.396807] LNetError: 41744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.405879] LNet: Removed LNI 192.168.202.134@tcp [ 1004.773191] Key type .llcrypt unregistered [ 1004.774950] Key type ._llcrypt unregistered [ 1018.848768] Key type ._llcrypt registered [ 1018.850246] Key type .llcrypt registered [ 1018.905894] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1019.452987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.539868] alg: No test for adler32 (adler32-zlib) [ 1020.458390] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1020.576172] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1022.175221] Key type lgssc registered [ 1022.799482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1027.305956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1027.320661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.560739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1030.697391] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1035.228302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.445873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1037.350749] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1039.929989] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1041.507730] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1044.610371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1044.750352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.991061] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1049.344586] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1053.990124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1064.419168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1064.422522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.429773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1065.447051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1065.453708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1065.469227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1067.196556] Lustre: lustre-OST0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 1067.198834] Lustre: Skipped 1 previous similar message [ 1069.562877] Lustre: server umount lustre-OST0000 complete [ 1085.922030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.928475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1085.932814] Lustre: Skipped 1 previous similar message [ 1090.583515] Lustre: server umount lustre-MDT0000 complete [ 1092.067836] LustreError: 43124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.076321] LustreError: 43124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1092.553425] LustreError: 43102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424309 with bad export cookie 17929230406009050747 [ 1092.554265] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1092.560392] LustreError: 43102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1098.781102] Lustre: server umount lustre-MDT0001 complete [ 1104.817758] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1106.448857] Key type lgssc unregistered [ 1106.625804] LNet: 46496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.630251] LNetError: 46496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.643561] LNet: Removed LNI 192.168.202.134@tcp [ 1107.136262] Key type .llcrypt unregistered [ 1107.138851] Key type ._llcrypt unregistered [ 1121.185142] Key type ._llcrypt registered [ 1121.186878] Key type .llcrypt registered [ 1121.245551] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1121.759961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1121.858946] alg: No test for adler32 (adler32-zlib) [ 1122.750872] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1122.888090] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1124.487208] Key type lgssc registered [ 1125.081192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1129.671560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1129.686836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.009269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.190586] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1137.870950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.054101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1139.973933] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1142.715420] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1144.757845] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1148.933397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1149.131723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1152.259916] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1155.007605] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1156.335421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1168.468053] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1170.020065] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1181.021517] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-2820h59m56s [ 1181.741667] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1182.889100] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 1183.202113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.207083] Lustre: Skipped 1 previous similar message [ 1183.902094] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 1183.906487] Lustre: Skipped 3 previous similar messages [ 1188.860878] Lustre: server umount lustre-MDT0000 complete [ 1190.047217] LustreError: 47869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.059296] LustreError: 47869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1190.628583] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1194.600622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.697305] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1194.879516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.910076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1195.884375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1197.004956] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1198.837276] Lustre: DEBUG MARKER: Wait for df (36041) ... [ 1199.623904] Lustre: DEBUG MARKER: done [ 1203.679942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1203.682921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.687936] Lustre: Skipped 1 previous similar message [ 1203.691396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1203.693798] Lustre: Skipped 3 previous similar messages [ 1205.218109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1205.224303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.242561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1206.245308] Lustre: Skipped 2 previous similar messages [ 1207.333642] Lustre: server umount lustre-OST0000 complete [ 1211.361883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1211.362325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.369997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1211.387016] Lustre: Skipped 1 previous similar message [ 1215.537388] Lustre: server umount lustre-MDT0000 complete [ 1217.504994] LustreError: 47867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.521358] LustreError: 47867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1217.686345] LustreError: 47853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424434 with bad export cookie 13775936131662793100 [ 1217.689256] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1217.694609] LustreError: 47853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1217.900339] Lustre: server umount lustre-MDT0001 complete [ 1223.880667] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1225.608642] Key type lgssc unregistered [ 1225.783255] LNet: 52447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.792829] LNetError: 52447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.803542] LNet: Removed LNI 192.168.202.134@tcp [ 1226.332114] Key type .llcrypt unregistered [ 1226.333249] Key type ._llcrypt unregistered [ 1248.819240] Key type ._llcrypt registered [ 1248.821216] Key type .llcrypt registered [ 1248.890123] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1249.605781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1249.615207] alg: No test for adler32 (adler32-zlib) [ 1250.532086] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1250.676907] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1252.295201] Key type lgssc registered [ 1252.925861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1257.258198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1257.270198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.424987] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1258.440489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1258.535547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.603057] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1265.410139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.470989] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1265.487895] Lustre: 54629:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1265.623247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1267.501561] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1269.802361] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1271.455706] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1274.630090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1274.743706] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1274.838263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.624293] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1280.069817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1280.286237] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1285.184102] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-2821h01m40s [ 1285.889301] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1289.129606] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1290.361949] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1290.389315] LustreError: 54635:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1310.895652] Lustre: lustre-MDT0000: Client cb93db5f-dafa-4c0d-8ccb-784f3bf120bb (at 192.168.202.34@tcp) reconnecting [ 1330.463132] LustreError: 54635:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1331.649097] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1333.087168] Lustre: DEBUG MARKER: done [ 1336.294242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.295937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.310954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.318123] Lustre: Skipped 1 previous similar message [ 1341.414799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.419028] Lustre: Skipped 1 previous similar message [ 1342.058670] Lustre: server umount lustre-OST0000 complete [ 1346.528596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.533181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.539224] Lustre: Skipped 2 previous similar messages [ 1346.551451] Lustre: Skipped 1 previous similar message [ 1350.254723] Lustre: server umount lustre-MDT0000 complete [ 1352.301744] LustreError: 54620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424568 with bad export cookie 4292676287606215825 [ 1352.307585] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1352.485232] Lustre: server umount lustre-MDT0001 complete [ 1358.404978] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1359.809712] Key type lgssc unregistered [ 1359.976797] LNet: 58594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.983236] LNetError: 58594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.992818] LNet: Removed LNI 192.168.202.134@tcp [ 1360.435098] Key type .llcrypt unregistered [ 1360.436933] Key type ._llcrypt unregistered [ 1380.168162] Key type ._llcrypt registered [ 1380.169836] Key type .llcrypt registered [ 1380.218678] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1380.722116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1380.742330] alg: No test for adler32 (adler32-zlib) [ 1381.648968] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1381.753440] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1383.359534] Key type lgssc registered [ 1383.839500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1385.252272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1385.266597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.404193] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1386.420649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1386.507756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.908257] Lustre: Failing over lustre-MDT0000 [ 1387.027141] Lustre: server umount lustre-MDT0000 complete [ 1389.585379] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:03:25 (1777424605) [ 1392.390743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.592192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.207398] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1398.298894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.348629] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1398.361685] Lustre: 61354:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1398.481439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1400.302787] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1402.901633] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1404.598582] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1407.911310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.010996] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1408.098833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.208621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1410.706062] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1413.140094] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1417.612990] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1424.352092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1424.355603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.363207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1424.873520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.874175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1424.883991] Lustre: Skipped 1 previous similar message [ 1424.892935] Lustre: Skipped 1 previous similar message [ 1426.025440] Lustre: server umount lustre-MDT0000 complete [ 1428.019847] LustreError: 61346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424644 with bad export cookie 5543436544085247367 [ 1428.022177] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1428.025657] LustreError: 61346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1428.252695] Lustre: server umount lustre-MDT0001 complete [ 1429.753456] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1431.801141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1436.036390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.114856] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4cee3e2a9d1ece5c [ 1438.126409] Lustre: MGC192.168.202.134@tcp: Connection restored to 0@lo (at 0@lo) [ 1438.294234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.336928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1440.179900] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1443.300424] LustreError: 62977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.301369] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.311787] LustreError: 62977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1444.583465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.863501] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1446.433081] Lustre: 60280:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777424646/real 1777424646] req@000000007c7f0509 x1863764775305088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777424662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1446.966405] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1449.502150] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1449.503108] Lustre: 60280:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777424650/real 1777424650] req@0000000046c729be x1863764775305216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777424666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1451.380880] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1454.048191] Lustre: 60277:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777424654/real 1777424654] req@00000000044273be x1863764775305472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777424670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1455.073401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1455.076739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.082136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.462746] Lustre: server umount lustre-MDT0000 complete [ 1461.216542] LustreError: 64778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.223408] LustreError: 64778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1462.164427] LustreError: 61344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424678 with bad export cookie 5543436544085249628 [ 1462.166818] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1462.170202] LustreError: 61344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1462.296948] Lustre: server umount lustre-MDT0001 complete [ 1465.249422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1469.638618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.611850] LustreError: 62976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.674742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.678478] Lustre: Skipped 1 previous similar message [ 1471.730412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1473.943140] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1477.089513] LustreError: 66182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1477.090463] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.109993] Lustre: Skipped 2 previous similar messages [ 1477.999194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.219787] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1482.335113] Lustre: 60278:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777424682/real 1777424682] req@00000000f9ec3610 x1863764775326336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777424698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1482.913518] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1483.235200] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1483.239521] Lustre: Skipped 1 previous similar message [ 1484.411536] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1486.312810] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1487.648252] Lustre: 60279:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777424687/real 1777424687] req@0000000044018390 x1863764775326592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777424703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1487.661901] Lustre: 60279:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1488.352994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1488.356719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.362906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1488.367641] Lustre: Skipped 3 previous similar messages [ 1492.770032] Lustre: server umount lustre-MDT0000 complete [ 1494.454562] LustreError: 61345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424711 with bad export cookie 5543436544085251007 [ 1494.457768] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.460058] LustreError: 61345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1494.572905] Lustre: server umount lustre-MDT0001 complete [ 1497.512717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1499.149248] Lustre: DEBUG MARKER: files compared the same [ 1510.469294] Lustre: server umount lustre-OST0000 complete [ 1518.610774] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1519.962932] Key type lgssc unregistered [ 1520.112849] LNet: 70264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.116439] LNetError: 70264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.127495] LNet: Removed LNI 192.168.202.134@tcp [ 1520.540844] Key type .llcrypt unregistered [ 1520.542349] Key type ._llcrypt unregistered [ 1531.209194] Key type ._llcrypt registered [ 1531.210831] Key type .llcrypt registered [ 1531.255043] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1531.616842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1531.641937] alg: No test for adler32 (adler32-zlib) [ 1532.522481] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1532.622017] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1534.223192] Key type lgssc registered [ 1534.637489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1537.715785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1537.726570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.918094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.415252] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug +malloc all [ 1544.070406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.211479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1545.783200] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug +malloc all [ 1548.064553] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1549.577120] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1552.559638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1552.699825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1555.004680] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug +malloc all [ 1557.434915] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1564.963980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1574.367734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1574.370287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.375294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1575.392767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1575.392871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1575.392912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1580.511924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1580.514576] Lustre: Skipped 1 previous similar message [ 1585.633601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1585.636238] Lustre: Skipped 1 previous similar message [ 1586.143122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1586.202034] Lustre: server umount lustre-OST0000 complete [ 1590.240396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1590.249295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.255345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.257776] Lustre: Skipped 1 previous similar message [ 1593.946135] Lustre: server umount lustre-MDT0000 complete [ 1595.748108] LustreError: 71622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424812 with bad export cookie 12082867102714519892 [ 1595.749575] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1595.752888] LustreError: 71622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1595.881508] Lustre: server umount lustre-MDT0001 complete [ 1600.234606] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1601.476124] Key type lgssc unregistered [ 1601.631637] LNet: 74838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.635892] LNetError: 74838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.647600] LNet: Removed LNI 192.168.202.134@tcp [ 1602.056049] Key type .llcrypt unregistered [ 1602.057743] Key type ._llcrypt unregistered [ 1612.218440] Key type ._llcrypt registered [ 1612.220160] Key type .llcrypt registered [ 1612.264316] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1612.605509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1612.648031] alg: No test for adler32 (adler32-zlib) [ 1613.525813] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1613.630769] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1615.223249] Key type lgssc registered [ 1615.688524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1619.254966] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1619.266456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1634.783400] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1639.903313] LustreError: 76189:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 1666.527168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.599468] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1670.601316] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1673.662416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1673.725347] Lustre: *** cfs_fail_loc=706, val=0*** [ 1673.879717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.926508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1675.437527] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1678.894416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.057607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1680.447813] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1682.493069] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1683.747087] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1686.498205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1686.499149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1686.502784] Lustre: Skipped 1 previous similar message [ 1686.506153] Lustre: Skipped 1 previous similar message [ 1690.701782] Lustre: server umount lustre-OST0000 complete [ 1694.688396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1694.691059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1694.696241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1696.736918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.737813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1696.743533] Lustre: Skipped 1 previous similar message [ 1698.133768] Lustre: server umount lustre-MDT0000 complete [ 1699.383021] LustreError: 76195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424915 with bad export cookie 9168221217010518694 [ 1699.385582] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1699.386751] LustreError: 76195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1699.491798] Lustre: server umount lustre-MDT0001 complete [ 1703.054633] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1704.052347] Key type lgssc unregistered [ 1704.189366] LNet: 79449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.192614] LNetError: 79449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.202331] LNet: Removed LNI 192.168.202.134@tcp [ 1704.507584] Key type .llcrypt unregistered [ 1704.509261] Key type ._llcrypt unregistered [ 1714.190616] Key type ._llcrypt registered [ 1714.192191] Key type .llcrypt registered [ 1714.235418] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1714.670397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1714.701065] alg: No test for adler32 (adler32-zlib) [ 1715.577159] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1715.664266] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1717.255163] Key type lgssc registered [ 1717.628748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1720.303051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1720.308860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.598940] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1725.072047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.236765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1726.453678] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1729.358910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1729.470761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1731.428636] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1734.511756] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1734.516159] Lustre: MGS: Client caae4baa-8a9d-4851-b6fb-be14670dd54e (at 0@lo) reconnecting [ 1734.618773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.661442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1735.943771] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1744.864115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1744.866444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.871284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1745.889751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.890338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1745.894612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1745.896609] Lustre: Skipped 1 previous similar message [ 1749.843132] Lustre: server umount lustre-OST0000 complete [ 1756.128591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1756.128774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.129346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1756.137594] Lustre: Skipped 1 previous similar message [ 1757.314635] Lustre: server umount lustre-MDT0000 complete [ 1758.750764] Lustre: server umount lustre-MDT0001 complete [ 1760.125141] Lustre: server umount MGS complete [ 1764.101622] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:09:39 (1777424979) [ 1773.607465] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 1776.947693] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1788.989812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.452599] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1794.809908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.898423] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1794.918397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1794.956443] Lustre: lustre-MDT0001: new disk, initializing [ 1794.986815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1794.999884] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1795.003106] Lustre: 88554:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1795.006185] LustreError: 88554:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1796.441819] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1800.942566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1801.029605] Lustre: lustre-OST0000: new disk, initializing [ 1801.031668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1801.052468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.171413] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1803.679893] Lustre: 89542:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000a2616fad x1863765125530368/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777425036 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1806.505664] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1808.895068] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1808.900481] Lustre: MGS: Client 73a44be6-4061-4ea9-bd02-f36ae1c88a66 (at 0@lo) reconnecting [ 1808.917923] Lustre: 87841:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1808.924652] LustreError: 88539:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1808.927917] LustreError: 88539:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1808.934126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1808.976378] Lustre: lustre-MDT0000: new disk, initializing [ 1809.045285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1810.079691] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1810.082935] Lustre: 90264:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b0391e9a x1863765125536640/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1777425042 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1810.563132] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1812.131792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1812.136779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1812.206875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1829.856538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1829.859917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.866965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1829.872325] Lustre: Skipped 1 previous similar message [ 1835.797775] Lustre: server umount lustre-OST0000 complete [ 1842.144780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1842.144964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.149441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.154264] Lustre: Skipped 2 previous similar messages [ 1842.158930] Lustre: Skipped 3 previous similar messages [ 1843.531142] Lustre: server umount lustre-MDT0000 complete [ 1851.549171] Lustre: server umount lustre-MDT0001 complete [ 1853.325268] Lustre: server umount MGS complete [ 1858.172599] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:11:13 (1777425073) [ 1864.993540] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1866.176499] Key type lgssc unregistered [ 1866.324403] LNet: 92726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.327653] LNetError: 92726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.335381] LNet: Removed LNI 192.168.202.134@tcp [ 1866.677656] Key type .llcrypt unregistered [ 1866.679429] Key type ._llcrypt unregistered [ 1873.746810] Key type ._llcrypt registered [ 1873.748106] Key type .llcrypt registered [ 1873.788438] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1874.129321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1874.172758] alg: No test for adler32 (adler32-zlib) [ 1875.025292] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1875.113840] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1876.695168] Key type lgssc registered [ 1877.053168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1879.665625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1879.672486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.834657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.153722] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1885.020294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.161495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1886.590833] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1888.490501] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1889.871870] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1892.553051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1892.688524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.726210] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1896.694367] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1906.022081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1911.265827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1911.268477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.276607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1913.170850] Lustre: server umount lustre-OST0000 complete [ 1915.872849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1915.876559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.881435] Lustre: Skipped 1 previous similar message [ 1915.883484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.886363] Lustre: Skipped 1 previous similar message [ 1916.655608] Lustre: server umount lustre-MDT0000 complete [ 1918.154065] LustreError: 93889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425134 with bad export cookie 12636350717720778148 [ 1918.156984] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1918.158323] LustreError: 93889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1918.271563] Lustre: server umount lustre-MDT0001 complete [ 1921.844838] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1922.929397] Key type lgssc unregistered [ 1923.067451] LNet: 97103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.070701] LNetError: 97103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.082322] LNet: Removed LNI 192.168.202.134@tcp [ 1923.391588] Key type .llcrypt unregistered [ 1923.392977] Key type ._llcrypt unregistered [ 1930.002123] Key type ._llcrypt registered [ 1930.003205] Key type .llcrypt registered [ 1930.047464] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1930.409629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.420744] alg: No test for adler32 (adler32-zlib) [ 1931.285906] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 1931.372996] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1932.959159] Key type lgssc registered [ 1933.331592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1935.621682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1935.621713] LustreError: 98146:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 1935.624434] Lustre: Skipped 1 previous similar message [ 1935.624445] LustreError: 98147:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 1935.631565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.632750] LustreError: 98146:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 1935.641169] LustreError: 98146:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1935.647035] LustreError: 98146:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1936.831267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.800875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.917161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1941.354858] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1942.964829] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing lsmod [ 1943.968446] LustreError: 99318:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 1943.968475] LustreError: 99317:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 1943.975141] LustreError: 99318:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1943.977769] LustreError: 99318:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1943.980823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1943.982490] LustreError: 99318:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1944.114948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.058159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1945.060302] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.061252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.067187] Lustre: Skipped 2 previous similar messages [ 1950.177718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1950.179860] Lustre: Skipped 4 previous similar messages [ 1955.297780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1955.299938] Lustre: Skipped 2 previous similar messages [ 1959.391124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1959.462793] Lustre: server umount lustre-MDT0000 complete [ 1960.418332] LustreError: 98184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.425817] LustreError: 98184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1960.907092] LustreError: 98160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425177 with bad export cookie 14073805789624931582 [ 1960.910516] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1960.911595] LustreError: 98160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1962.463282] LustreError: 98615:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000123e5a22 x1863765350958208/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1962.520150] Lustre: server umount lustre-MDT0001 complete [ 1973.787290] Lustre: server umount lustre-OST0000 complete [ 1976.721624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.902034] LustreError: 100388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.929594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.248179] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1981.286017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.419679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1982.770470] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1984.492771] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1985.666155] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1987.947658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1989.981863] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1991.956362] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2002.405780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2006.495779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2006.498054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.502113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2007.521152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2007.521439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.993471] Lustre: server umount lustre-OST0000 complete [ 2012.127908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2012.130154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.134337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.136194] Lustre: Skipped 2 previous similar messages [ 2017.424694] Lustre: server umount lustre-MDT0000 complete [ 2017.760851] LustreError: 100407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.765693] LustreError: 100407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2018.757329] LustreError: 100370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425235 with bad export cookie 14073805789624932254 [ 2018.760452] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2018.760961] LustreError: 100370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2018.886498] Lustre: server umount lustre-MDT0001 complete [ 2022.467147] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2023.501969] Key type lgssc unregistered [ 2023.633341] LNet: 103585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.635809] LNetError: 103585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.642268] LNet: Removed LNI 192.168.202.134@tcp [ 2023.969034] Key type .llcrypt unregistered [ 2023.970856] Key type ._llcrypt unregistered [ 2033.345349] Key type ._llcrypt registered [ 2033.346595] Key type .llcrypt registered [ 2033.389599] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2033.721968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2033.779351] alg: No test for adler32 (adler32-zlib) [ 2034.638264] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 2034.723975] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2036.311162] Key type lgssc registered [ 2036.673211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2039.387385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2039.393589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2040.551792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.988393] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2044.868136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.991485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2046.303271] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2048.167159] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2049.367818] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2051.720895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2051.840183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.960928] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2055.698175] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2066.214045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2073.120299] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2073.869287] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2075.616551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2075.619685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.624715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2076.641273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2076.642150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.642559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2080.843488] Lustre: server umount lustre-OST0000 complete [ 2085.855956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2085.859192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.864275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2085.867391] Lustre: Skipped 1 previous similar message [ 2088.406634] Lustre: server umount lustre-MDT0000 complete [ 2089.799846] LustreError: 107177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425306 with bad export cookie 16431817421871969586 [ 2089.803519] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2089.804676] LustreError: 107177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2089.922711] Lustre: server umount lustre-MDT0001 complete [ 2094.337502] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2095.521268] Key type lgssc unregistered [ 2095.687431] LNet: 108262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2095.690285] LNetError: 108262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2095.700535] LNet: Removed LNI 192.168.202.134@tcp [ 2096.015365] Key type .llcrypt unregistered [ 2096.017148] Key type ._llcrypt unregistered [ 2103.507617] Key type ._llcrypt registered [ 2103.509874] Key type .llcrypt registered [ 2103.553198] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2104.048239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2104.067472] alg: No test for adler32 (adler32-zlib) [ 2104.951312] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 2105.075662] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2106.671171] Key type lgssc registered [ 2107.156789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2110.602119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2110.609107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.841082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2113.373518] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2116.700061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.841957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2118.352118] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2120.751895] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2122.335955] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2125.011020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2125.191515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.300641] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2129.362872] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2136.492955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2142.688137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2142.690751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.694928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2146.785560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2146.787557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2146.791246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.623586] Lustre: server umount lustre-OST0000 complete [ 2151.904714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.905413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2151.909056] Lustre: Skipped 1 previous similar message [ 2151.911706] Lustre: Skipped 1 previous similar message [ 2157.024157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2157.027514] Lustre: Skipped 2 previous similar messages [ 2162.144785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2162.147127] Lustre: Skipped 1 previous similar message [ 2164.191110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2164.252536] Lustre: server umount lustre-MDT0000 complete [ 2165.482125] LustreError: 111074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425382 with bad export cookie 1281358116558465231 [ 2165.485565] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.485637] LustreError: 111074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2165.587890] Lustre: server umount lustre-MDT0001 complete [ 2168.921574] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2169.913231] Key type lgssc unregistered [ 2170.050605] LNet: 112660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.054284] LNetError: 112660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.064297] LNet: Removed LNI 192.168.202.134@tcp [ 2170.379438] Key type .llcrypt unregistered [ 2170.380540] Key type ._llcrypt unregistered [ 2179.618932] Key type ._llcrypt registered [ 2179.620459] Key type .llcrypt registered [ 2179.658146] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2179.959914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2180.002959] alg: No test for adler32 (adler32-zlib) [ 2180.882484] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 2180.973031] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2182.559126] Key type lgssc registered [ 2182.931021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2185.574992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2185.581213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.733491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.103037] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2191.462542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.577501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2193.061312] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2195.239390] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2196.618146] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2199.272385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2199.414819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.544670] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2203.662411] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2211.690366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2216.783279] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2218.207446] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2218.209839] Lustre: Skipped 2 previous similar messages [ 2219.827211] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2219.830274] Lustre: Skipped 2 previous similar messages [ 2222.078231] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2222.092561] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2228.128644] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.34@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2228.133670] Lustre: Skipped 2 previous similar messages [ 2228.736515] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.34@tcp 192.168.0.[2,10]@tcp [ 2236.083772] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.34/24@tcp in log lustre-MDT0000 [ 2236.087918] Lustre: Skipped 2 previous similar messages [ 2245.621488] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.34/24@tcp [ 2245.623922] Lustre: Skipped 1 previous similar message [ 2248.161833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2248.165892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.172460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2252.769050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.770056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2252.773148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2252.774559] Lustre: Skipped 1 previous similar message [ 2254.027750] Lustre: server umount lustre-OST0000 complete [ 2257.889750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.890306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2257.895810] Lustre: Skipped 1 previous similar message [ 2261.468594] Lustre: server umount lustre-MDT0000 complete [ 2262.766221] LustreError: 115651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425479 with bad export cookie 9745304620750251143 [ 2262.769775] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.770114] LustreError: 115651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2262.879650] Lustre: server umount lustre-MDT0001 complete [ 2266.373585] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2267.428694] Key type lgssc unregistered [ 2267.567556] LNet: 118783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2267.570347] LNetError: 118783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2267.577412] LNet: Removed LNI 192.168.202.134@tcp [ 2267.895897] Key type .llcrypt unregistered [ 2267.897044] Key type ._llcrypt unregistered [ 2277.923257] Key type ._llcrypt registered [ 2277.924226] Key type .llcrypt registered [ 2277.953829] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2278.258802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.267778] alg: No test for adler32 (adler32-zlib) [ 2279.129190] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 2279.210527] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2280.791194] Key type lgssc registered [ 2281.126019] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2283.736477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2283.742678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.908077] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2284.914180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.918781] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.34/24@tcp [ 2286.219203] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2289.161885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.263046] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2289.272720] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.34/24@tcp [ 2289.288414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2290.564276] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2292.316710] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2293.532905] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2295.880949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2295.988489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.783282] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2299.537966] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2310.310228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2319.840127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2319.843400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.847551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2320.865993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2320.866529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2320.867168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.869393] Lustre: Skipped 1 previous similar message [ 2324.703773] Lustre: server umount lustre-OST0000 complete [ 2330.592202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2330.594441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.599568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.245520] Lustre: server umount lustre-MDT0000 complete [ 2333.626501] LustreError: 120284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425550 with bad export cookie 15555384596338615954 [ 2333.630809] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2333.634338] LustreError: 120284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2333.756026] Lustre: server umount lustre-MDT0001 complete [ 2337.487617] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2338.499549] Key type lgssc unregistered [ 2338.612476] LNet: 123558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.614993] LNetError: 123558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2338.622312] LNet: Removed LNI 192.168.202.134@tcp [ 2338.907398] Key type .llcrypt unregistered [ 2338.908785] Key type ._llcrypt unregistered [ 2347.691072] Key type ._llcrypt registered [ 2347.692436] Key type .llcrypt registered [ 2347.735495] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 2353.810280] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2354.253179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2354.260639] alg: No test for adler32 (adler32-zlib) [ 2355.108584] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 2355.192770] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2356.775134] Key type lgssc registered [ 2357.117781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2368.040107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2368.044934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.129319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2369.142595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2369.177246] Lustre: lustre-MDT0000: new disk, initializing [ 2369.202232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.211478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2370.479692] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2374.602626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.641668] Lustre: 126896:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2374.666299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2374.668134] Lustre: Skipped 1 previous similar message [ 2374.701984] Lustre: lustre-MDT0001: new disk, initializing [ 2374.734627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2374.746544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2374.750323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2376.035763] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2379.187889] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2380.365151] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2382.714576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2382.811638] Lustre: lustre-OST0000: new disk, initializing [ 2382.813343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2382.839269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.480247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2383.484146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2383.522501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2384.724018] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2387.871480] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2389.176486] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2389.243284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2390.502982] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2390.576309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2394.081389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2394.081888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.085244] LustreError: Skipped 1 previous similar message [ 2394.091380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2394.094273] Lustre: Skipped 1 previous similar message [ 2397.331369] Lustre: server umount lustre-OST0000 complete [ 2399.201088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2399.201610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2399.205224] Lustre: Skipped 2 previous similar messages [ 2404.320376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.322134] Lustre: Skipped 2 previous similar messages [ 2404.760357] Lustre: server umount lustre-MDT0000 complete [ 2405.996287] LustreError: 126887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425622 with bad export cookie 8609278355551146379 [ 2405.998396] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2406.000479] LustreError: 126887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2406.117496] Lustre: server umount lustre-MDT0001 complete [ 2409.026418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2409.196359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.440112] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2413.347276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.470220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2414.735866] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2416.579642] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2417.727216] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2420.024851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2421.963981] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2423.606656] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2424.892081] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2434.129907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2435.395770] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2435.461311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2438.809215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2438.867231] Lustre: lustre-OST0001: new disk, initializing [ 2438.869404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2438.893628] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2438.896887] Lustre: Skipped 1 previous similar message [ 2440.847189] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2444.156272] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2446.846487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2446.850230] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2446.877987] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2447.283237] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2448.730213] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2448.816102] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2450.087705] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e4d8892b800.ost_server_uuid 50 [ 2450.589883] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e4d8892b800.ost_server_uuid in FULL state after 0 sec [ 2457.061548] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2457.063942] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.068605] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2457.071393] Lustre: Skipped 1 previous similar message [ 2458.429516] Lustre: server umount lustre-OST0001 complete [ 2460.127733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2460.130432] LustreError: Skipped 1 previous similar message [ 2460.132340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.137189] Lustre: Skipped 1 previous similar message [ 2462.177444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2462.177853] LustreError: 133720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.179612] Lustre: Skipped 2 previous similar messages [ 2462.186507] LustreError: 133720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2465.805512] Lustre: server umount lustre-OST0000 complete [ 2467.297488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.301759] Lustre: Skipped 2 previous similar messages [ 2472.415956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2472.417805] Lustre: Skipped 3 previous similar messages [ 2473.118021] Lustre: server umount lustre-MDT0000 complete [ 2474.383179] LustreError: 133766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425690 with bad export cookie 8609278355551147247 [ 2474.386454] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2474.387028] LustreError: 133766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2474.496286] Lustre: server umount lustre-MDT0001 complete [ 2480.884370] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2481.973854] Key type lgssc unregistered [ 2482.104392] LNet: 136621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2482.107127] LNetError: 136621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2482.113316] LNet: Removed LNI 192.168.202.134@tcp [ 2482.416323] Key type .llcrypt unregistered [ 2482.418034] Key type ._llcrypt unregistered [ 2495.950514] Key type ._llcrypt registered [ 2495.951565] Key type .llcrypt registered [ 2495.987631] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 2502.099768] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2502.411528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.492605] alg: No test for adler32 (adler32-zlib) [ 2503.367962] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 2503.454043] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2505.039145] Key type lgssc registered [ 2505.415810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2518.009735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2518.018528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.113601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2519.127885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2519.163372] Lustre: lustre-MDT0000: new disk, initializing [ 2519.189640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.196372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2520.543261] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2524.893706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.939264] Lustre: 140716:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2524.962321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2524.964877] Lustre: Skipped 1 previous similar message [ 2525.000534] Lustre: lustre-MDT0001: new disk, initializing [ 2525.036853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2525.051055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2525.056882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2526.558383] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2529.998969] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2531.362881] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2533.951972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2534.051907] Lustre: lustre-OST0000: new disk, initializing [ 2534.054662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2534.085581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.250151] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2539.763741] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2544.122472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2544.126116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2544.153788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2547.111219] Lustre: Failing over lustre-OST0000 [ 2547.186278] Lustre: server umount lustre-OST0000 complete [ 2560.303786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.424988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2560.436292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.443885] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2561.701436] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2562.571146] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2563.951344] Lustre: Failing over lustre-MDT0000 [ 2564.055385] Lustre: server umount lustre-MDT0000 complete [ 2564.959171] Lustre: 138943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777425765/real 1777425765] req@000000002b580393 x1863765951801984/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1777425781 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2565.600704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.605467] Lustre: Skipped 2 previous similar messages [ 2570.079144] Lustre: 138941:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777425770/real 1777425770] req@000000002909d853 x1863765951803904/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777425786 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2570.720930] LustreError: 140727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.729527] LustreError: 140727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2571.934066] LustreError: 140723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.839123] Lustre: 138942:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777425776/real 1777425776] req@000000002fae91c6 x1863765951805056/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777425792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2575.840200] LustreError: 140728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.857614] LustreError: 140728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2577.685490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.741464] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.746420] Lustre: 140720:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.134@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2577.831855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2577.860658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.861173] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2577.861376] Lustre: lustre-OST0000: Recovery over after 0:16, of 3 clients 3 recovered and 0 were evicted. [ 2577.893716] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2579.442835] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2582.175572] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2583.012038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2583.014200] Lustre: Skipped 1 previous similar message [ 2583.022732] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2583.039648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2588.128693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2588.131088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.131461] LustreError: Skipped 1 previous similar message [ 2588.135355] Lustre: Skipped 1 previous similar message [ 2588.135948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2588.141795] Lustre: Skipped 1 previous similar message [ 2590.230301] Lustre: server umount lustre-OST0000 complete [ 2593.248531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.248905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2593.249118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.252191] Lustre: Skipped 2 previous similar messages [ 2597.586977] Lustre: server umount lustre-MDT0000 complete [ 2598.369699] LustreError: 140723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.375625] LustreError: 140723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2598.924056] LustreError: 142459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425815 with bad export cookie 15343338104374950768 [ 2598.927338] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.929191] LustreError: 142459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2599.046348] Lustre: server umount lustre-MDT0001 complete [ 2602.690100] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2603.661733] Key type lgssc unregistered [ 2603.789316] LNet: 145945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2603.791759] LNetError: 145945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2603.800257] LNet: Removed LNI 192.168.202.134@tcp [ 2604.093703] Key type .llcrypt unregistered [ 2604.094934] Key type ._llcrypt unregistered [ 2624.043766] Key type ._llcrypt registered [ 2624.045042] Key type .llcrypt registered [ 2624.089164] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2624.504606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2624.544971] alg: No test for adler32 (adler32-zlib) [ 2625.410831] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 2625.485545] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2627.063147] Key type lgssc registered [ 2627.411016] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2629.925860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2629.931638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.077738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.361460] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2635.474972] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2637.449805] LustreError: 147472:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2637.453172] LustreError: 147472:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2637.456602] LustreError: 147472:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2637.576618] Lustre: server umount lustre-MDT0000 complete [ 2642.395716] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 2645.612821] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2657.382267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.484519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2657.494231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2657.532613] Lustre: lustre-MDT0000: new disk, initializing [ 2657.555617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.562812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2658.848360] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2662.901701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.939801] Lustre: 150620:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2662.961763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2662.963919] Lustre: Skipped 1 previous similar message [ 2663.001105] Lustre: lustre-MDT0001: new disk, initializing [ 2663.035952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2663.047539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2663.050642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2664.247600] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2667.238336] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2668.343264] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2670.619388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.738729] Lustre: lustre-OST0000: new disk, initializing [ 2670.740480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2670.769994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.576754] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2673.844181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2673.847110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2673.874095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2675.607962] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2677.435951] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2678.023350] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:24:53 (1777425893) [ 2745.311419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2745.313669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.316903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2745.823462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.823648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2745.828148] Lustre: Skipped 1 previous similar message [ 2747.420292] Lustre: server umount lustre-MDT0000 complete [ 2748.571369] LustreError: 151399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425965 with bad export cookie 13123929260208773557 [ 2748.572234] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2748.575601] LustreError: 151399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2750.943411] LustreError: 151812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.943436] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.948191] LustreError: 151812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2750.951840] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2750.956219] Lustre: Skipped 1 previous similar message [ 2753.951451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2753.951451] LustreError: 152965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.710980] Lustre: server umount lustre-MDT0001 complete [ 2770.399115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2770.445622] Lustre: server umount lustre-OST0000 complete [ 2773.532530] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 2776.450427] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2788.218064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2788.305912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2788.318336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2788.359161] Lustre: lustre-MDT0000: new disk, initializing [ 2788.382345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2788.387976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2789.740484] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2794.073970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.116609] Lustre: 156843:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2794.140490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2794.142110] Lustre: Skipped 1 previous similar message [ 2794.182587] Lustre: lustre-MDT0001: new disk, initializing [ 2794.237107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2794.240975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2795.596071] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2798.785459] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2800.042554] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2802.354500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.460651] Lustre: lustre-OST0000: new disk, initializing [ 2802.462579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2802.492878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.495324] Lustre: Skipped 1 previous similar message [ 2803.130212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2803.134736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2803.174716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2804.374082] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2807.451254] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2808.783368] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2808.870352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2810.227758] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2810.311212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2813.408763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2813.412529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.414288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2813.418028] Lustre: Skipped 1 previous similar message [ 2813.423204] Lustre: Skipped 1 previous similar message [ 2823.652159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2823.654101] Lustre: Skipped 3 previous similar messages [ 2825.183167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2825.236261] Lustre: server umount lustre-OST0000 complete [ 2828.768648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.775018] Lustre: Skipped 1 previous similar message [ 2832.600390] Lustre: server umount lustre-MDT0000 complete [ 2833.838109] LustreError: 156836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426050 with bad export cookie 13123929260208932373 [ 2833.840905] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2833.841487] LustreError: 156836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2833.956304] Lustre: server umount lustre-MDT0001 complete [ 2836.756272] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:27:32 (1777426052) [ 2842.035072] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 2845.271272] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2855.939859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2856.018083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2856.026796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2856.063285] Lustre: lustre-MDT0000: new disk, initializing [ 2856.084722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.092500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2857.283304] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2861.291192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.334359] Lustre: 163159:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2861.433045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2862.757105] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2865.822135] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2867.047838] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2869.326371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2870.391937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2870.429506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2871.378195] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2874.492735] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2880.993075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2880.993709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2880.995845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2880.995853] Lustre: Skipped 2 previous similar messages [ 2880.996748] LustreError: Skipped 2 previous similar messages [ 2881.007525] Lustre: Skipped 1 previous similar message [ 2882.452722] Lustre: server umount lustre-OST0000 complete [ 2897.887123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2897.947237] Lustre: server umount lustre-MDT0000 complete [ 2899.260352] LustreError: 164909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426115 with bad export cookie 13123929260208933241 [ 2899.263125] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2899.264059] LustreError: 164909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2899.392533] Lustre: server umount lustre-MDT0001 complete [ 2902.259962] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:28:38 (1777426118) [ 2907.669408] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 2910.881699] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2922.642744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.732865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2922.744391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2922.746324] Lustre: Skipped 3 previous similar messages [ 2922.783389] Lustre: lustre-MDT0000: new disk, initializing [ 2922.784624] Lustre: Skipped 2 previous similar messages [ 2922.807251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.809685] Lustre: Skipped 2 previous similar messages [ 2922.815160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2922.817624] Lustre: Skipped 2 previous similar messages [ 2924.119349] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2928.317225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.358679] Lustre: 169075:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2928.456977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2929.783171] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2932.979850] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2934.219520] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2936.468249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2938.474753] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2939.287833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2941.547660] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2944.481901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.483251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2944.485949] Lustre: Skipped 2 previous similar messages [ 2944.489461] LustreError: Skipped 1 previous similar message [ 2944.491489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2944.494703] Lustre: Skipped 8 previous similar messages [ 2949.588733] Lustre: server umount lustre-OST0000 complete [ 2954.207858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2956.949432] Lustre: server umount lustre-MDT0000 complete [ 2958.161051] LustreError: 169069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426174 with bad export cookie 13123929260208934312 [ 2958.164477] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2958.164591] LustreError: 169069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2958.274804] Lustre: server umount lustre-MDT0001 complete [ 2961.881764] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2962.914627] Key type lgssc unregistered [ 2963.035368] LNet: 172557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2963.037959] LNetError: 172557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2963.044330] LNet: Removed LNI 192.168.202.134@tcp [ 2963.355120] Key type .llcrypt unregistered [ 2963.356191] Key type ._llcrypt unregistered [ 2972.269774] Key type ._llcrypt registered [ 2972.270949] Key type .llcrypt registered [ 2972.308751] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2972.651648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2972.705031] alg: No test for adler32 (adler32-zlib) [ 2973.561183] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 2973.644159] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2975.223182] Key type lgssc registered [ 2975.607937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2978.237805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2978.244754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.396308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2980.709900] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2983.411745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.540131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2984.875234] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2986.588815] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2987.704022] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2989.968463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2990.095975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2992.076117] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2993.909339] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3003.430977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3018.723173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.724950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.726753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3018.726787] Lustre: Skipped 1 previous similar message [ 3022.293583] Lustre: server umount lustre-OST0000 complete [ 3023.841055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.841824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.845147] Lustre: Skipped 1 previous similar message [ 3023.847794] Lustre: Skipped 1 previous similar message [ 3028.960282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3028.962495] Lustre: Skipped 1 previous similar message [ 3029.714424] Lustre: server umount lustre-MDT0000 complete [ 3030.963323] LustreError: 174651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426247 with bad export cookie 15958953544926233438 [ 3030.966295] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3030.968274] LustreError: 174651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3031.081679] Lustre: server umount lustre-MDT0001 complete [ 3034.812353] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3035.872888] Key type lgssc unregistered [ 3035.990355] LNet: 177133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3035.992667] LNetError: 177133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3035.999278] LNet: Removed LNI 192.168.202.134@tcp [ 3036.321147] Key type .llcrypt unregistered [ 3036.322948] Key type ._llcrypt unregistered [ 3045.505952] Key type ._llcrypt registered [ 3045.506978] Key type .llcrypt registered [ 3045.544797] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3045.879932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3045.946375] alg: No test for adler32 (adler32-zlib) [ 3046.810128] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 3046.892745] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3048.471168] Key type lgssc registered [ 3048.847107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3051.475420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3051.482452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.678028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.031671] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3056.941335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.066784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3058.385924] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3060.205706] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3061.477524] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3063.687324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3063.806296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.659602] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3067.500453] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3078.185588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3082.208048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3082.210274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.214312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3083.232978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3083.233701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3083.235767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.608512] Lustre: server umount lustre-OST0000 complete [ 3088.395141] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e4d82d3f000.ost_server_uuid 50 [ 3088.868126] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e4d82d3f000.ost_server_uuid in DISCONN state after 0 sec [ 3089.370362] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3099.368103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.368729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3099.372607] Lustre: Skipped 1 previous similar message [ 3099.375637] Lustre: Skipped 2 previous similar messages [ 3104.735995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3104.738088] Lustre: Skipped 2 previous similar messages [ 3105.489824] Lustre: server umount lustre-MDT0000 complete [ 3106.701397] LustreError: 179224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426323 with bad export cookie 15526825882936021837 [ 3106.704221] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.704809] LustreError: 179224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3106.815431] Lustre: server umount lustre-MDT0001 complete [ 3109.615254] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:32:05 (1777426325) [ 3111.775919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.921193] LustreError: 182037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3111.929152] LustreError: 182037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3111.944961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.196253] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3115.965180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.090052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3117.412087] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3119.265633] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3120.482283] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3122.770861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3124.868696] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3126.630303] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3128.961616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3129.037781] Lustre: lustre-OST0001: new disk, initializing [ 3129.039354] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3129.073623] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3129.076053] Lustre: Skipped 1 previous similar message [ 3131.021725] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3134.128474] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3134.464698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3134.468140] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3134.496516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3136.550260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3141.604956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.605255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.609648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3141.612013] LustreError: Skipped 1 previous similar message [ 3142.527072] Lustre: server umount lustre-OST0000 complete [ 3144.568212] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3144.643848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3144.869386] LustreError: 184138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.992338] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3146.069162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3146.723399] LustreError: 183630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.728991] LustreError: 183630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3149.982251] LustreError: 184138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.101825] LustreError: 183632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.107397] LustreError: 183632:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3156.963775] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.965499] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3156.968703] Lustre: Skipped 2 previous similar messages [ 3156.972086] Lustre: Skipped 2 previous similar messages [ 3162.196611] Lustre: server umount lustre-OST0001 complete [ 3167.200917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3167.200951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.208469] Lustre: Skipped 1 previous similar message [ 3169.619550] Lustre: server umount lustre-MDT0000 complete [ 3170.853203] LustreError: 182019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426387 with bad export cookie 15526825882936022901 [ 3170.855859] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3170.857247] LustreError: 182019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3170.987243] Lustre: server umount lustre-MDT0001 complete [ 3178.849283] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:33:14 (1777426394) [ 3180.986327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.076652] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3181.087361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3181.157916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3182.519042] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3185.467361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.501541] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3185.513633] Lustre: 187809:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3187.052405] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3188.938273] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3190.149607] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3192.395183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3192.461181] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3193.050081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3194.337153] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3196.159552] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3198.528382] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3198.557235] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3200.514894] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3201.875234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3202.312512] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3203.552052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3203.552272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.555252] LustreError: Skipped 1 previous similar message [ 3203.560157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3203.562883] Lustre: Skipped 5 previous similar messages [ 3209.277536] Lustre: server umount lustre-OST0000 complete [ 3212.259222] LustreError: 189430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.264896] LustreError: 189430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3222.499343] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3222.501748] LustreError: Skipped 1 previous similar message [ 3226.063135] Lustre: server umount lustre-OST0001 complete [ 3233.498632] Lustre: server umount lustre-MDT0000 complete [ 3234.804452] LustreError: 187799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426451 with bad export cookie 15526825882936024287 [ 3234.807800] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3234.809185] LustreError: 187799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3234.941884] Lustre: server umount lustre-MDT0001 complete [ 3243.203059] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:34:19 (1777426459) [ 3248.470250] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 3251.446341] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3263.591076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.681855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3263.693241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3263.731199] Lustre: lustre-MDT0000: new disk, initializing [ 3263.754565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.756775] Lustre: Skipped 3 previous similar messages [ 3263.761981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3265.054312] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3269.157913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.195944] Lustre: 195378:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3269.217830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3269.220475] Lustre: Skipped 1 previous similar message [ 3269.258232] Lustre: lustre-MDT0001: new disk, initializing [ 3269.300108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3269.304883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3270.553397] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3273.651111] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3274.847331] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3277.128189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.239482] Lustre: lustre-OST0000: new disk, initializing [ 3277.242221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3279.170532] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3282.315522] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3283.714975] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3283.959091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3283.963184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3283.989553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3284.807844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3286.115377] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3286.184968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3289.056857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3289.058197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.060509] LustreError: Skipped 1 previous similar message [ 3289.064449] Lustre: Skipped 5 previous similar messages [ 3289.065295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3289.071216] Lustre: Skipped 10 previous similar messages [ 3292.948084] Lustre: server umount lustre-OST0000 complete [ 3300.381268] Lustre: server umount lustre-MDT0000 complete [ 3301.586142] LustreError: 195371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426518 with bad export cookie 15526825882936026058 [ 3301.589118] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3301.589801] LustreError: 195371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3301.706449] Lustre: server umount lustre-MDT0001 complete [ 3304.599030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3304.753832] LustreError: 199193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.761336] LustreError: 199193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3306.109355] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3309.077992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.485194] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3312.340089] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3313.582896] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3315.833609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3317.867180] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3319.694395] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3321.133752] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3331.382749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3332.673052] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3332.755203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3334.624603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3334.627982] LustreError: Skipped 1 previous similar message [ 3339.556122] Lustre: server umount lustre-OST0000 complete [ 3341.662772] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3341.731294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3343.025337] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3366.818568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3366.956584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.959435] Lustre: Skipped 5 previous similar messages [ 3368.897407] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3370.634391] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3371.161403] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3374.157330] Lustre: server umount lustre-OST0000 complete [ 3375.376178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.376842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.379748] Lustre: Skipped 6 previous similar messages [ 3375.383856] Lustre: Skipped 7 previous similar messages [ 3377.495631] Lustre: server umount lustre-MDT0000 complete [ 3378.763252] LustreError: 199176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426595 with bad export cookie 15526825882936026926 [ 3378.766385] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3378.767748] LustreError: 199176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3378.892638] Lustre: server umount lustre-MDT0001 complete [ 3381.642990] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:36:37 (1777426597) [ 3383.893237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.064730] LustreError: 204723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.374686] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3388.688660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.097777] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3391.811697] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3393.042949] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3395.284812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3397.337103] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3399.337249] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3400.763098] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3409.092117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3410.480648] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3410.556132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3412.778841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3412.864813] Lustre: lustre-OST0001: new disk, initializing [ 3412.866545] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3414.850819] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3418.008395] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3419.457051] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3422.745792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3422.751941] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3422.797544] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3423.642161] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3424.969931] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3425.042638] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3427.812052] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3427.814294] LustreError: Skipped 2 previous similar messages [ 3431.742535] Lustre: server umount lustre-OST0001 complete [ 3433.640474] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3433.711616] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3434.965695] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3435.041557] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3448.294239] LustreError: 206315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.300527] LustreError: 206315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3458.744483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.672552] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3462.440114] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3463.094843] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3469.949950] Lustre: server umount lustre-OST0001 complete [ 3488.275238] Lustre: server umount lustre-OST0000 complete [ 3495.721611] Lustre: server umount lustre-MDT0000 complete [ 3497.022284] LustreError: 204705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426713 with bad export cookie 15526825882936027857 [ 3497.024779] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3497.026881] LustreError: 204705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3503.254945] Lustre: server umount lustre-MDT0001 complete [ 3511.668469] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:38:47 (1777426727) [ 3513.888028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.984079] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3513.993364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3514.069734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.071710] Lustre: Skipped 5 previous similar messages [ 3515.321371] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3518.200318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3518.231644] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3518.240730] Lustre: 212747:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3519.749410] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3521.596319] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3522.773458] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3525.169672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3525.240141] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3527.148882] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3528.929523] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3538.175507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3538.199129] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3540.373578] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3542.456989] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3543.000784] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3544.278844] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3544.344230] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3545.637283] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3545.709241] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3546.866177] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e4d8b92c800.ost_server_uuid 50 [ 3547.358254] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e4d8b92c800.ost_server_uuid in FULL state after 0 sec [ 3547.700880] Lustre: Permanently deactivating lustre-OST0001 [ 3547.707482] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3548.501531] Lustre: Permanently reactivating lustre-OST0001 [ 3550.178557] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.182704] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3550.184528] Lustre: Skipped 7 previous similar messages [ 3550.189242] Lustre: Skipped 16 previous similar messages [ 3555.640121] Lustre: server umount lustre-OST0001 complete [ 3557.855936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3557.858362] LustreError: Skipped 3 previous similar messages [ 3571.167142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3571.221370] Lustre: server umount lustre-OST0000 complete [ 3578.600834] Lustre: server umount lustre-MDT0000 complete [ 3579.908207] LustreError: 212740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426796 with bad export cookie 15526825882936029117 [ 3579.912424] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3579.913217] LustreError: 212740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3580.057716] Lustre: server umount lustre-MDT0001 complete [ 3588.086552] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:40:03 (1777426803) [ 3590.648534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.753081] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3590.765136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3590.767934] Lustre: Skipped 5 previous similar messages [ 3592.186205] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3595.503056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.535089] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3595.545179] Lustre: 218847:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3597.090096] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3598.845962] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3600.124191] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3602.240514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3602.318831] Lustre: Permanently deactivating lustre-OST0000 [ 3602.323572] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3602.688248] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3604.195265] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3606.045986] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3608.241848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3610.171214] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3611.473453] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3612.001461] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3612.599812] Lustre: Permanently reactivating lustre-OST0000 [ 3618.812327] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3618.814970] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3618.818351] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3618.840402] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3618.846896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3632.006243] Lustre: server umount lustre-OST0001 complete [ 3632.101400] LustreError: 220467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.106062] LustreError: 220467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 3639.369206] Lustre: server umount lustre-OST0000 complete [ 3640.915279] Lustre: server umount lustre-MDT0000 complete [ 3642.158836] LustreError: 218839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426858 with bad export cookie 15526825882936030769 [ 3642.162044] LustreError: 218839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3642.163095] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3642.281237] Lustre: server umount lustre-MDT0001 complete [ 3645.717092] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3646.691066] Key type lgssc unregistered [ 3646.815365] LNet: 223247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3646.817720] LNetError: 223247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3646.827352] LNet: Removed LNI 192.168.202.134@tcp [ 3647.128338] Key type .llcrypt unregistered [ 3647.129350] Key type ._llcrypt unregistered [ 3655.305641] Key type ._llcrypt registered [ 3655.307228] Key type .llcrypt registered [ 3655.346393] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3655.684945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3655.720951] alg: No test for adler32 (adler32-zlib) [ 3656.585949] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 3656.668773] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3658.247112] Key type lgssc registered [ 3658.601465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3661.891485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3661.897952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.057697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3664.289571] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3667.135664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.189507] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3667.202892] Lustre: 224663:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3667.209955] Lustre: Permanently deactivating lustre-MDT0001 [ 3667.215892] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3667.337840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3668.717252] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3670.666664] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3671.823063] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3673.069591] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3674.079881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3674.189566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3676.053275] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3677.281151] LustreError: 226790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3677.284473] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3677.286348] LustreError: 226790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3677.769367] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3680.019829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3680.107820] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3682.059599] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3683.909137] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3685.356301] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3688.427678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3688.429898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3689.722463] Lustre: Permanently reactivating lustre-MDT0001 [ 3689.731797] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3689.733968] Lustre: Skipped 2 previous similar messages [ 3691.172207] Lustre: lustre-MDT0001: Client 6f834e98-edd8-4c6b-b87f-4991f55a77b5 (at 192.168.202.34@tcp) reconnecting [ 3692.986714] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3693.591557] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3695.105079] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3695.109293] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3695.111835] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3695.115682] LustreError: 228147:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3695.116088] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3695.118583] LustreError: 228147:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3695.124176] LustreError: 228147:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -108 [ 3695.127137] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3695.127137] [ 3695.129818] LustreError: 224665:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3695.318673] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3695.417147] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3696.754933] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3696.824954] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3697.578100] Lustre: Permanently deactivating lustre-MDT0001 [ 3697.583490] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3697.585437] Lustre: Skipped 2 previous similar messages [ 3703.808166] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3708.384050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3708.387029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.391650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3708.899477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3708.902712] Lustre: Skipped 2 previous similar messages [ 3712.753511] Lustre: server umount lustre-MDT0000 complete [ 3713.999271] LustreError: 227773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426930 with bad export cookie 734242453053386380 [ 3714.003421] LustreError: 227773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3714.004580] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3714.015704] LustreError: 224669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3714.021719] LustreError: 224669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3714.123726] Lustre: server umount lustre-MDT0001 complete [ 3721.470332] Lustre: server umount lustre-OST0000 complete [ 3725.185670] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:42:21 (1777426941) [ 3729.540322] Lustre: server umount lustre-OST0001 complete [ 3731.680792] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 3734.719098] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3746.688654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3746.802280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3746.806650] Lustre: Skipped 2 previous similar messages [ 3746.818834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3746.858711] Lustre: lustre-MDT0000: new disk, initializing [ 3746.883657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3746.890172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3748.324823] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3752.827714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.879810] Lustre: 232358:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3752.907553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3752.909306] Lustre: Skipped 1 previous similar message [ 3752.952106] Lustre: lustre-MDT0001: new disk, initializing [ 3753.003574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3753.007636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3754.290079] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3757.954069] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3759.664208] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3762.820110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3762.922241] Lustre: lustre-OST0000: new disk, initializing [ 3762.925264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3762.952201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3762.955017] Lustre: Skipped 1 previous similar message [ 3763.849703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3763.855913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3763.914925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3765.195528] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3768.568473] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3769.283634] LustreError: 232363:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3773.813863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3773.909939] Lustre: lustre-OST0001: new disk, initializing [ 3773.913702] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3776.344646] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3779.375108] LustreError: 232363:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3779.385482] LustreError: 233695:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3779.586408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3779.590707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3779.633909] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3779.789921] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3780.388587] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3789.479143] LustreError: 233695:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3792.556224] Lustre: lustre-OST0001: Not available for connect from 192.168.202.34@tcp (stopping) [ 3794.914348] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3794.918082] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.923101] Lustre: Skipped 3 previous similar messages [ 3794.926045] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3794.930050] Lustre: Skipped 1 previous similar message [ 3796.430763] Lustre: server umount lustre-OST0001 complete [ 3800.037643] LustreError: 234085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3800.048048] LustreError: 234085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3805.155450] LustreError: 235521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.161804] LustreError: 235521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3810.276287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3810.277466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.278303] LustreError: 234083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.278900] LustreError: Skipped 2 previous similar messages [ 3810.283084] Lustre: Skipped 1 previous similar message [ 3810.284192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3810.295851] Lustre: Skipped 1 previous similar message [ 3820.512710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3820.514585] Lustre: Skipped 3 previous similar messages [ 3820.514669] LustreError: 234085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.520095] LustreError: 234085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3823.071183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3823.161829] Lustre: server umount lustre-OST0000 complete [ 3825.636196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3825.644424] Lustre: Skipped 2 previous similar messages [ 3831.014242] Lustre: server umount lustre-MDT0000 complete [ 3832.317670] LustreError: 232350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427048 with bad export cookie 734242453053389194 [ 3832.324281] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3832.324998] LustreError: 232350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3838.513555] Lustre: server umount lustre-MDT0001 complete [ 3842.280493] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3843.321900] Key type lgssc unregistered [ 3843.457448] LNet: 237007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3843.461764] LNetError: 237007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3843.470460] LNet: Removed LNI 192.168.202.134@tcp [ 3843.777460] Key type .llcrypt unregistered [ 3843.778548] Key type ._llcrypt unregistered [ 3858.934520] Key type ._llcrypt registered [ 3858.935465] Key type .llcrypt registered [ 3858.975601] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3859.439096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3859.500126] alg: No test for adler32 (adler32-zlib) [ 3860.379331] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 3860.473783] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3862.063161] Key type lgssc registered [ 3862.504958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3865.256072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3865.263213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.382793] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3866.391579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3866.466236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3867.891520] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3871.381066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.425950] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3871.438710] Lustre: 239174:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3871.555775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3872.973689] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3874.827752] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3876.389584] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3878.794914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3878.872315] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3878.931995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3880.919351] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3882.659955] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3885.090741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3931.108033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3931.109668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.110603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3931.110896] LustreError: Skipped 1 previous similar message [ 3931.115384] Lustre: Skipped 1 previous similar message [ 3932.957567] Lustre: server umount lustre-OST0000 complete [ 3934.412171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3938.570381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3938.688904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3940.597334] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3942.358530] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3950.950371] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3950.953506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3950.953664] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3950.960290] Lustre: Skipped 1 previous similar message [ 3976.673919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.674161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3976.679636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3976.682785] Lustre: Skipped 2 previous similar messages [ 3981.567780] Lustre: server umount lustre-OST0000 complete [ 3984.351855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3984.355323] LustreError: Skipped 1 previous similar message [ 3984.357182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.362045] Lustre: Skipped 1 previous similar message [ 3984.364265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3986.912495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3986.913213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.081794] Lustre: server umount lustre-MDT0000 complete [ 3990.313343] LustreError: 239897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427206 with bad export cookie 2168359265722597512 [ 3990.317742] LustreError: 239897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3990.317765] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3990.443169] Lustre: server umount lustre-MDT0001 complete [ 3994.005551] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3994.980423] Key type lgssc unregistered [ 3995.098409] LNet: 244120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3995.101338] LNetError: 244120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3995.110344] LNet: Removed LNI 192.168.202.134@tcp [ 3995.408981] Key type .llcrypt unregistered [ 3995.410713] Key type ._llcrypt unregistered [ 4004.717477] Key type ._llcrypt registered [ 4004.718548] Key type .llcrypt registered [ 4004.761881] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4005.073517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4005.093741] alg: No test for adler32 (adler32-zlib) [ 4005.953761] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4006.043445] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4007.631208] Key type lgssc registered [ 4008.025688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4010.729936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4010.735385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.910931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4013.236418] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4016.320442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4016.449908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4017.887254] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4019.707748] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4020.875633] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4023.154145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4023.273158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.120584] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4026.931228] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4036.582658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4046.818143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4046.818756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.819453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4046.826447] Lustre: Skipped 1 previous similar message [ 4050.957609] Lustre: server umount lustre-OST0000 complete [ 4052.448238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4052.451987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4052.458470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4052.461674] Lustre: Skipped 1 previous similar message [ 4057.056554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4057.057495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.342539] Lustre: server umount lustre-MDT0000 complete [ 4059.683264] LustreError: 246212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427276 with bad export cookie 12654954451288164882 [ 4059.688414] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4059.689299] LustreError: 246212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4059.817676] Lustre: server umount lustre-MDT0001 complete [ 4063.655666] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4064.819066] Key type lgssc unregistered [ 4064.944561] LNet: 249464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4064.947628] LNetError: 249464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4064.960475] LNet: Removed LNI 192.168.202.134@tcp [ 4065.292685] Key type .llcrypt unregistered [ 4065.293634] Key type ._llcrypt unregistered [ 4072.932511] Key type ._llcrypt registered [ 4072.934181] Key type .llcrypt registered [ 4072.989979] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4073.556321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4073.602386] alg: No test for adler32 (adler32-zlib) [ 4074.504433] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4074.595128] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4076.175193] Key type lgssc registered [ 4076.539147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4079.647285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4079.655934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4080.834525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.134276] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4085.200730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4085.349650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4086.758725] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4088.560552] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4089.831938] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4092.299224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4092.449385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.307762] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4096.078273] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4105.766079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4110.817258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4110.819725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.821567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4110.825190] Lustre: Skipped 1 previous similar message [ 4111.961198] Lustre: server umount lustre-OST0000 complete [ 4115.364621] Lustre: server umount lustre-MDT0000 complete [ 4115.936913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4115.937114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.937819] LustreError: 250641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.937826] LustreError: 250641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4115.957285] Lustre: Skipped 1 previous similar message [ 4116.704302] LustreError: 250627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427333 with bad export cookie 13930050245779718031 [ 4116.707388] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4116.707596] LustreError: 250627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4116.831282] Lustre: server umount lustre-MDT0001 complete [ 4120.509899] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4121.485265] Key type lgssc unregistered [ 4121.593386] LNet: 253990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4121.595721] LNetError: 253990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4121.604269] LNet: Removed LNI 192.168.202.134@tcp [ 4121.881407] Key type .llcrypt unregistered [ 4121.883148] Key type ._llcrypt unregistered [ 4130.502463] Key type ._llcrypt registered [ 4130.503485] Key type .llcrypt registered [ 4130.537013] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4130.918798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4130.927816] alg: No test for adler32 (adler32-zlib) [ 4131.785083] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4131.865160] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4133.447204] Key type lgssc registered [ 4134.108585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4137.346862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4137.354241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.526418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4140.025862] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4143.130112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.303266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4144.894860] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4147.007450] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4148.374674] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4150.856195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4150.982797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.736401] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4154.460616] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4164.330101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4173.792698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4173.797984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.807404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4174.816638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4174.817966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4174.818023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.826572] Lustre: Skipped 1 previous similar message [ 4179.096083] Lustre: server umount lustre-OST0000 complete [ 4184.543864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4184.545994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.549594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4186.732436] Lustre: server umount lustre-MDT0000 complete [ 4188.008401] LustreError: 255347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427404 with bad export cookie 15993429063860964833 [ 4188.011132] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4188.011521] LustreError: 255347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4188.135897] Lustre: server umount lustre-MDT0001 complete [ 4192.588650] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4193.755957] Key type lgssc unregistered [ 4193.887527] LNet: 259339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4193.890498] LNetError: 259339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4193.901472] LNet: Removed LNI 192.168.202.134@tcp [ 4194.183747] Key type .llcrypt unregistered [ 4194.185047] Key type ._llcrypt unregistered [ 4201.638757] Key type ._llcrypt registered [ 4201.640109] Key type .llcrypt registered [ 4201.676718] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4202.009474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4202.063076] alg: No test for adler32 (adler32-zlib) [ 4202.913224] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4203.002355] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4204.591170] Key type lgssc registered [ 4205.118727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4208.566719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4208.573349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.786050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.296472] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4215.128873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4215.295189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4216.774388] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4218.715985] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4219.933933] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4222.331450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4222.507544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4224.578655] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4226.666333] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4234.787690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4244.962729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4244.963125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.966199] LustreError: Skipped 1 previous similar message [ 4244.969812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4244.972931] Lustre: Skipped 1 previous similar message [ 4246.478582] Lustre: server umount lustre-OST0000 complete [ 4250.081940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.082654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4250.085759] Lustre: Skipped 2 previous similar messages [ 4255.200549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4255.203326] Lustre: Skipped 2 previous similar messages [ 4260.320550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4260.323650] Lustre: Skipped 1 previous similar message [ 4262.367106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4262.430430] Lustre: server umount lustre-MDT0000 complete [ 4263.705661] LustreError: 261275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427480 with bad export cookie 5401579696550096161 [ 4263.709620] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4263.710271] LustreError: 261275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4263.851856] Lustre: server umount lustre-MDT0001 complete [ 4267.391886] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4268.495285] Key type lgssc unregistered [ 4268.623528] LNet: 263899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4268.626193] LNetError: 263899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4268.633461] LNet: Removed LNI 192.168.202.134@tcp [ 4268.944613] Key type .llcrypt unregistered [ 4268.945912] Key type ._llcrypt unregistered [ 4272.458113] Key type ._llcrypt registered [ 4272.458977] Key type .llcrypt registered [ 4272.495965] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4278.231110] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4284.393730] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4284.884929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4284.895146] alg: No test for adler32 (adler32-zlib) [ 4285.772690] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4285.877686] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4287.463172] Key type lgssc registered [ 4287.884811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4299.825096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4299.833358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.948723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4300.959757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4301.004626] Lustre: lustre-MDT0000: new disk, initializing [ 4301.031647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.037673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4302.423294] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4307.301702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.343140] Lustre: 267369:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4307.367188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4307.369124] Lustre: Skipped 1 previous similar message [ 4307.411431] Lustre: lustre-MDT0001: new disk, initializing [ 4307.443839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4307.455609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4307.459772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4308.771742] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4311.854435] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4313.093177] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4315.503957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4315.614071] Lustre: lustre-OST0000: new disk, initializing [ 4315.616288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4315.644730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4315.952321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4315.954976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4315.998651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4317.473209] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4320.653350] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4322.002931] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4322.082332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4323.463015] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4323.548723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4326.373950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.375238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4326.375707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4326.377233] Lustre: Skipped 1 previous similar message [ 4326.379479] LustreError: Skipped 1 previous similar message [ 4330.338564] Lustre: server umount lustre-OST0000 complete [ 4333.024031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4333.026455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4333.030054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4333.031657] Lustre: Skipped 1 previous similar message [ 4336.608494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4336.609372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.816552] Lustre: server umount lustre-MDT0000 complete [ 4339.057380] LustreError: 269109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427555 with bad export cookie 10126423491993971015 [ 4339.061557] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4339.062138] LustreError: 269109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4339.193757] Lustre: server umount lustre-MDT0001 complete [ 4342.067228] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:52:38 (1777427558) [ 4344.383362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.581474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4345.966821] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4349.111485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.281874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4350.612398] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4352.447941] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4353.625175] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4355.982345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4358.049133] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4359.894105] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4374.495902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4374.498553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.503180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4374.504911] Lustre: Skipped 1 previous similar message [ 4379.473459] Lustre: server umount lustre-OST0000 complete [ 4383.714545] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4383.719420] LustreError: Skipped 1 previous similar message [ 4383.722041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.729041] Lustre: Skipped 1 previous similar message [ 4383.732043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4383.735742] Lustre: Skipped 2 previous similar messages [ 4387.097824] Lustre: server umount lustre-MDT0000 complete [ 4388.310504] LustreError: 271352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427604 with bad export cookie 10126423491993971883 [ 4388.314371] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4388.314820] LustreError: 271352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4388.320455] LustreError: 272091:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4388.455203] Lustre: server umount lustre-MDT0001 complete [ 4391.967562] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4393.011506] Key type lgssc unregistered [ 4393.141498] LNet: 274574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4393.143851] LNetError: 274574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4393.155321] LNet: Removed LNI 192.168.202.134@tcp [ 4393.455359] Key type .llcrypt unregistered [ 4393.456479] Key type ._llcrypt unregistered [ 4402.241027] Key type ._llcrypt registered [ 4402.242185] Key type .llcrypt registered [ 4402.276884] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4408.632145] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4409.096795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4409.121537] alg: No test for adler32 (adler32-zlib) [ 4409.986362] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4410.071494] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4411.655114] Key type lgssc registered [ 4412.003026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4426.940021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4426.948335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4428.048485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4428.059800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4428.098410] Lustre: lustre-MDT0000: new disk, initializing [ 4428.122984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.128692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4429.423016] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4433.573322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.620551] Lustre: 278336:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4433.650729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4433.653357] Lustre: Skipped 1 previous similar message [ 4433.694356] Lustre: lustre-MDT0001: new disk, initializing [ 4433.730617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4433.744790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4433.749646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4435.020899] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4438.138353] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4439.461086] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4441.829357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4441.955315] Lustre: lustre-OST2710: new disk, initializing [ 4441.957612] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4441.987974] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4443.445378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4443.449751] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4443.476961] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4444.004236] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4447.189739] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4449.470898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4449.540733] Lustre: lustre-OST03e8: new disk, initializing [ 4449.542488] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4449.575204] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4451.564208] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4451.840986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4451.843929] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4451.875872] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4453.333240] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4455.908254] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4455.988356] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4457.371291] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4457.452761] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4458.856283] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4458.936774] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4460.301916] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4460.384130] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4462.051308] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.053371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.055653] Lustre: Skipped 2 previous similar messages [ 4462.057659] Lustre: Skipped 1 previous similar message [ 4467.169182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.172231] Lustre: Skipped 2 previous similar messages [ 4467.553405] Lustre: server umount lustre-MDT0000 complete [ 4468.724158] LustreError: 279115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427685 with bad export cookie 4844003964614403993 [ 4468.727281] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4468.728631] LustreError: 279115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4468.847570] Lustre: server umount lustre-MDT0001 complete [ 4479.499645] Lustre: server umount lustre-OST2710 complete [ 4491.033077] Lustre: server umount lustre-OST03e8 complete [ 4497.194936] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4500.675413] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4512.747061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.862660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4512.880763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4512.925273] Lustre: lustre-MDT0000: new disk, initializing [ 4512.963003] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.973222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4514.451623] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4518.837705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.892502] Lustre: 286050:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4519.016696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4520.289888] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4523.355142] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4524.568892] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4527.103063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4527.207567] Lustre: lustre-OST0000: new disk, initializing [ 4527.208934] Lustre: Skipped 1 previous similar message [ 4527.210567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4527.212123] Lustre: Skipped 2 previous similar messages [ 4527.239646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.241514] Lustre: Skipped 1 previous similar message [ 4529.126730] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4532.219945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4532.222169] Lustre: Skipped 1 previous similar message [ 4532.223747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4532.257187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4532.416358] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4534.122519] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4534.223470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4535.900242] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4535.973604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4537.314483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4537.314698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.318184] LustreError: Skipped 1 previous similar message [ 4537.324087] Lustre: Skipped 1 previous similar message [ 4537.327303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4537.331365] Lustre: Skipped 4 previous similar messages [ 4538.712897] Lustre: server umount lustre-OST0000 complete [ 4542.432843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4542.433496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4542.437536] Lustre: Skipped 1 previous similar message [ 4542.440708] Lustre: Skipped 1 previous similar message [ 4546.165288] Lustre: server umount lustre-MDT0000 complete [ 4547.498067] LustreError: 286042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427764 with bad export cookie 4844003964614405659 [ 4547.500691] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4547.501072] LustreError: 286042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4547.616569] Lustre: server umount lustre-MDT0001 complete [ 4550.593267] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:56:06 (1777427766) [ 4551.195850] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4551.739795] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:56:07 (1777427767) [ 4552.700449] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4562.433196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4562.576734] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4562.589870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4562.680900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.056420] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4567.047185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.079539] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4567.088357] Lustre: 291475:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4568.469237] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4570.275759] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4571.506431] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4586.785847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4586.872365] Lustre: Denying initial registration attempt from nid 192.168.202.134@tcp, specified as failover [ 4586.875810] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4586.878310] LustreError: 293084:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -99 [ 4586.880281] LustreError: 293084:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 4586.881910] LustreError: 293084:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 4586.904876] Lustre: server umount lustre-OST0000 complete [ 4586.906079] LustreError: 293084:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4593.119957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4593.123103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.129780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4595.496999] Lustre: server umount lustre-MDT0000 complete [ 4596.763897] LustreError: 291465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427813 with bad export cookie 4844003964614406527 [ 4596.769843] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4596.770445] LustreError: 291465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4596.890901] Lustre: server umount lustre-MDT0001 complete [ 4600.957621] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4602.058541] Key type lgssc unregistered [ 4602.197565] LNet: 294186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4602.201357] LNetError: 294186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4602.210535] LNet: Removed LNI 192.168.202.134@tcp [ 4602.528080] Key type .llcrypt unregistered [ 4602.529187] Key type ._llcrypt unregistered [ 4606.177857] Key type ._llcrypt registered [ 4606.178869] Key type .llcrypt registered [ 4606.221820] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4606.607160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4606.670053] alg: No test for adler32 (adler32-zlib) [ 4607.536170] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4607.647100] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4609.239150] Key type lgssc registered [ 4609.642756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4644.733291] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4647.584234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4647.590054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4648.695955] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4648.706446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4648.769675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4650.120515] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4652.951898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4652.983607] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4652.993150] Lustre: 296900:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4653.091241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4654.420987] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4656.179394] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4657.399897] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4672.803755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4672.889132] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4672.959148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4674.842090] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4676.652218] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4677.407447] Lustre: server umount lustre-OST0000 complete [ 4683.232755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4683.233464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4683.237221] Lustre: Skipped 1 previous similar message [ 4684.827554] Lustre: server umount lustre-MDT0000 complete [ 4686.065365] LustreError: 296891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427902 with bad export cookie 4381121470782766515 [ 4686.069152] LustreError: 296891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4686.069349] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4686.188221] Lustre: server umount lustre-MDT0001 complete [ 4689.588602] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4690.611640] Key type lgssc unregistered [ 4690.735352] LNet: 300112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4690.737747] LNetError: 300112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4690.746388] LNet: Removed LNI 192.168.202.134@tcp [ 4691.063672] Key type .llcrypt unregistered [ 4691.065033] Key type ._llcrypt unregistered [ 4700.182333] Key type ._llcrypt registered [ 4700.184012] Key type .llcrypt registered [ 4700.224601] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4700.533921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4700.549389] alg: No test for adler32 (adler32-zlib) [ 4701.418304] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4701.511443] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4703.103144] Key type lgssc registered [ 4703.452394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4706.092912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4706.100598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4707.277566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4708.653202] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4711.481876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.624407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4712.921796] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4714.715147] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4715.933266] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4720.226410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4720.349166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4722.206727] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4723.967911] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4736.993375] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.994076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4736.999202] Lustre: Skipped 1 previous similar message [ 4739.750349] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 4739.752360] Lustre: Skipped 2 previous similar messages [ 4742.113806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4742.115617] Lustre: Skipped 2 previous similar messages [ 4742.694625] Lustre: server umount lustre-MDT0000 complete [ 4743.931241] LustreError: 302203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427960 with bad export cookie 7075799531411886898 [ 4743.934446] LustreError: 302203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4743.935066] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4744.861514] LustreError: 301485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4744.868680] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [ 4744.870284] LustreError: 301485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4747.231991] LustreError: 301484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.232293] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4747.240834] Lustre: Skipped 1 previous similar message [ 4748.255826] LustreError: 302844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4749.981553] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [ 4749.984887] Lustre: Skipped 2 previous similar messages [ 4750.117673] Lustre: server umount lustre-MDT0001 complete [ 4752.290454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4755.170819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4768.671522] LustreError: 300799:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000979c615e x1863768256568192/t0(0) o250->MGC192.168.202.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4768.797180] LustreError: 303090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4768.801993] LustreError: 303090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4768.826933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4768.851605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4770.113499] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4772.896897] LustreError: 304763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4772.904052] LustreError: 304763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4772.918639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.054597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4774.285610] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4776.042789] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4778.216620] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4780.624806] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4781.752805] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4782.282052] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4783.590150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.591215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4783.592224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4783.595053] Lustre: Skipped 1 previous similar message [ 4789.657726] Lustre: server umount lustre-OST0000 complete [ 4793.825257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4793.825284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.834087] Lustre: Skipped 1 previous similar message [ 4797.027267] Lustre: server umount lustre-MDT0000 complete [ 4798.274337] LustreError: 301469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428014 with bad export cookie 7075799531411894052 [ 4798.279956] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4798.401947] Lustre: server umount lustre-MDT0001 complete [ 4802.015990] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4803.086469] Key type lgssc unregistered [ 4803.215362] LNet: 307192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4803.217801] LNetError: 307192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4803.223356] LNet: Removed LNI 192.168.202.134@tcp [ 4803.507723] Key type .llcrypt unregistered [ 4803.509020] Key type ._llcrypt unregistered [ 4812.837066] Key type ._llcrypt registered [ 4812.838087] Key type .llcrypt registered [ 4812.877762] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4813.340710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4813.351709] alg: No test for adler32 (adler32-zlib) [ 4814.210520] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4814.307419] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4815.887107] Key type lgssc registered [ 4816.284466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4818.954127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4818.960236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.127482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4821.431086] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4824.269810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4824.387144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4825.654230] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4827.414377] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4828.574843] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4829.663988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4829.666784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.672936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4833.760766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4833.761965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4835.751964] Lustre: server umount lustre-MDT0000 complete [ 4836.968197] LustreError: 309958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428053 with bad export cookie 11503720360300149853 [ 4836.971291] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4836.972081] LustreError: 309958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4837.089613] Lustre: server umount lustre-MDT0001 complete [ 4841.783645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.891736] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4841.901662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4841.971218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4843.238616] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4846.035944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.068439] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4846.078128] Lustre: 310929:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4846.172166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4847.453601] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4849.205230] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4850.430791] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4855.010187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4855.087584] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4855.091119] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4855.094159] LustreError: 312595:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 4855.096568] LustreError: 312595:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 4855.098567] LustreError: 312595:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 4855.119608] Lustre: server umount lustre-OST0000 complete [ 4855.121042] LustreError: 312595:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4859.874185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4859.947485] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4860.015715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4860.317448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4861.835742] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4863.543126] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4866.042038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4866.072145] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4866.104570] Lustre: lustre-OST0001: new disk, initializing [ 4866.107652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4867.936638] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4869.572578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4869.576736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4869.613191] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4869.633815] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4874.724703] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4874.724978] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4874.725849] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4874.725854] Lustre: Skipped 1 previous similar message [ 4874.728435] Lustre: Skipped 1 previous similar message [ 4874.735041] LustreError: Skipped 1 previous similar message [ 4876.670278] Lustre: server umount lustre-OST0001 complete [ 4879.842836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.842935] LustreError: 313061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.844078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4879.844088] Lustre: Skipped 1 previous similar message [ 4879.848147] Lustre: Skipped 1 previous similar message [ 4879.851996] LustreError: 313061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4884.962218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4884.963410] LustreError: 313061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.964044] Lustre: Skipped 2 previous similar messages [ 4884.971758] LustreError: 313061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4890.082759] LustreError: 314709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.090186] LustreError: 314709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4892.127148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4892.187985] Lustre: server umount lustre-OST0000 complete [ 4895.200769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.201566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4895.204543] Lustre: Skipped 1 previous similar message [ 4895.207717] Lustre: Skipped 3 previous similar messages [ 4899.547935] Lustre: server umount lustre-MDT0000 complete [ 4900.320045] LustreError: 310939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4900.327731] LustreError: 310939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4900.756118] LustreError: 310920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428117 with bad export cookie 11503720360300150392 [ 4900.758949] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4900.759526] LustreError: 310920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4900.884249] Lustre: server umount lustre-MDT0001 complete [ 4904.612126] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4905.746969] Key type lgssc unregistered [ 4905.884550] LNet: 315846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4905.888307] LNetError: 315846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4905.899544] LNet: Removed LNI 192.168.202.134@tcp [ 4906.217712] Key type .llcrypt unregistered [ 4906.218934] Key type ._llcrypt unregistered [ 4936.284317] Key type ._llcrypt registered [ 4936.285463] Key type .llcrypt registered [ 4936.327671] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4942.503298] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4943.058280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4943.065529] alg: No test for adler32 (adler32-zlib) [ 4943.915517] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 4943.999526] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4945.583171] Key type lgssc registered [ 4945.994500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4957.638103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4957.643765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4958.734855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4958.746883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4958.787569] Lustre: lustre-MDT0000: new disk, initializing [ 4958.812606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4958.819639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4959.996485] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4963.979930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.024065] Lustre: 320714:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4964.051680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4964.053370] Lustre: Skipped 1 previous similar message [ 4964.091073] Lustre: lustre-MDT0001: new disk, initializing [ 4964.128165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4964.139356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4964.142679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4965.477103] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4968.413756] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4969.617865] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4971.875089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4972.004829] Lustre: lustre-OST0000: new disk, initializing [ 4972.007705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4972.043349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4973.914647] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4973.952634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4973.955926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4973.992183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4977.025742] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4978.378981] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4978.462943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4979.741073] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4979.817607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4984.290197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4984.290910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4984.294518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4984.297267] Lustre: Skipped 1 previous similar message [ 4984.300318] Lustre: Skipped 1 previous similar message [ 4986.581645] Lustre: server umount lustre-OST0000 complete [ 4989.409709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.410901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4989.413353] Lustre: Skipped 1 previous similar message [ 4994.528049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4994.529668] Lustre: Skipped 2 previous similar messages [ 4999.648124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4999.651576] Lustre: Skipped 1 previous similar message [ 5002.207105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5002.270916] Lustre: server umount lustre-MDT0000 complete [ 5003.644467] LustreError: 320706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428220 with bad export cookie 6074415638684636872 [ 5003.647756] LustreError: 320706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5003.647774] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5003.767064] Lustre: server umount lustre-MDT0001 complete [ 5006.532731] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:03:42 (1777428222) [ 5008.950283] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:03:44 (1777428224) [ 5011.793341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.004965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5013.348024] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5016.198063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5016.324378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5017.615831] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5019.432398] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5020.608990] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5022.830859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5024.716949] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5026.503845] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5045.695734] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5046.265923] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5046.809284] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5047.372103] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5047.992030] Lustre: Failing over lustre-MDT0000 [ 5048.204621] Lustre: server umount lustre-MDT0000 complete [ 5052.383917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5052.386158] LustreError: Skipped 1 previous similar message [ 5052.387433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.391552] LustreError: 325060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.395829] LustreError: 325060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5052.897911] LustreError: 325057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.902237] LustreError: 325057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5057.693232] LustreError: 325056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5061.030063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.068771] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5061.173245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5061.175021] Lustre: Skipped 1 previous similar message [ 5061.203931] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5062.482067] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5062.812902] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5066.220146] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5066.229687] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5066.249927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5067.560275] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5068.126499] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5069.982134] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5070.637314] Lustre: Failing over lustre-MDT0000 [ 5070.806138] Lustre: server umount lustre-MDT0000 complete [ 5071.329363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5071.330598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.331136] LustreError: 325056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5071.331143] LustreError: 325056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5071.343129] Lustre: Skipped 3 previous similar messages [ 5076.448317] LustreError: 325060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5076.452900] LustreError: 325060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5083.806398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.851788] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5083.950736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.976566] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5085.219864] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5088.413378] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5089.261313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5089.263850] Lustre: Skipped 2 previous similar messages [ 5089.272817] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5089.291739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5090.512729] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5091.140788] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5094.369028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5094.370540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.372928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.375215] Lustre: Skipped 2 previous similar messages [ 5094.377782] Lustre: Skipped 2 previous similar messages [ 5099.610476] Lustre: server umount lustre-MDT0000 complete [ 5100.820135] LustreError: 326676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428317 with bad export cookie 6074415638684639553 [ 5100.823343] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5100.823344] LustreError: 326676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5100.948621] Lustre: server umount lustre-MDT0001 complete [ 5112.411771] Lustre: server umount lustre-OST0000 complete [ 5117.568403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.737409] LustreError: 330856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.742185] LustreError: 330856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5117.764809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5119.156812] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5122.019062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.593169] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5125.490684] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5126.693867] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5128.884378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5130.935602] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5132.626256] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5144.363498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5149.665708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5149.668390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.669325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5149.669332] Lustre: Skipped 3 previous similar messages [ 5149.675015] Lustre: Skipped 2 previous similar messages [ 5155.535493] Lustre: server umount lustre-OST0000 complete [ 5157.856277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5170.144029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5170.145722] Lustre: Skipped 9 previous similar messages [ 5171.167203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5171.238304] Lustre: server umount lustre-MDT0000 complete [ 5172.418472] LustreError: 330837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428388 with bad export cookie 6074415638684640274 [ 5172.421313] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5172.424153] LustreError: 330837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5172.526073] Lustre: server umount lustre-MDT0001 complete [ 5176.054552] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5177.053267] Key type lgssc unregistered [ 5177.159394] LNet: 334058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5177.162570] LNetError: 334058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5177.174340] LNet: Removed LNI 192.168.202.134@tcp [ 5177.462040] Key type .llcrypt unregistered [ 5177.463301] Key type ._llcrypt unregistered [ 5186.630160] Key type ._llcrypt registered [ 5186.631220] Key type .llcrypt registered [ 5186.666509] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5186.950587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5186.962467] alg: No test for adler32 (adler32-zlib) [ 5187.818363] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5187.903820] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5189.487134] Key type lgssc registered [ 5189.846341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5192.406930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5192.414308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.571741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.807179] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5197.467908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.593735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5198.765789] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5200.444261] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5201.666488] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5203.852564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5203.936833] LustreError: 337125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5203.941204] LustreError: 337125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5203.969855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5205.790817] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5206.389970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5207.524435] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5213.151929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5213.153926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.157480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5213.665934] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.672830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5213.674449] Lustre: Skipped 1 previous similar message [ 5218.455464] Lustre: server umount lustre-MDT0000 complete [ 5218.785061] LustreError: 335535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5218.793041] LustreError: 335535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5219.686144] LustreError: 335515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428436 with bad export cookie 4672574336360854179 [ 5219.689926] LustreError: 335515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5219.690078] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5219.808306] Lustre: server umount lustre-MDT0001 complete [ 5231.256774] Lustre: server umount lustre-OST0000 complete [ 5235.511945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.665377] LustreError: 338906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.688624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5237.023447] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5239.838743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.975829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5241.361736] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5243.036268] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5244.143838] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5246.274932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5247.398390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5248.143943] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5249.741646] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5259.745511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5259.747123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5259.751328] Lustre: Skipped 3 previous similar messages [ 5259.753026] Lustre: Skipped 1 previous similar message [ 5262.544934] Lustre: server umount lustre-MDT0000 complete [ 5263.698146] LustreError: 339623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428480 with bad export cookie 4672574336360855278 [ 5263.700800] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5263.701215] LustreError: 339623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5263.801808] Lustre: server umount lustre-MDT0001 complete [ 5275.221043] Lustre: server umount lustre-OST0000 complete [ 5280.052956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.196978] LustreError: 342475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.217326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5280.219183] Lustre: Skipped 1 previous similar message [ 5281.424902] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5284.171193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5285.594872] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5287.314220] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5288.515995] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5290.769451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5292.763426] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5294.514559] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5305.256818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5314.527891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5314.531686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.538280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5314.539970] Lustre: Skipped 1 previous similar message [ 5315.553480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5317.390838] Lustre: server umount lustre-OST0000 complete [ 5320.159861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5320.162242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.167611] Lustre: Skipped 1 previous similar message [ 5320.170265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5320.171946] Lustre: Skipped 2 previous similar messages [ 5324.708792] Lustre: server umount lustre-MDT0000 complete [ 5325.792246] LustreError: 342494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.796402] LustreError: 342494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5325.977849] LustreError: 342456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428542 with bad export cookie 4672574336360856286 [ 5325.983284] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5326.107282] Lustre: server umount lustre-MDT0001 complete [ 5329.712744] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5330.777353] Key type lgssc unregistered [ 5330.909368] LNet: 345674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5330.911834] LNetError: 345674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5330.920410] LNet: Removed LNI 192.168.202.134@tcp [ 5331.205300] Key type .llcrypt unregistered [ 5331.206617] Key type ._llcrypt unregistered [ 5340.729196] Key type ._llcrypt registered [ 5340.730268] Key type .llcrypt registered [ 5340.764407] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5341.167298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5341.189781] alg: No test for adler32 (adler32-zlib) [ 5342.094718] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5342.175986] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5343.759131] Key type lgssc registered [ 5344.079310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5346.439769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5346.445911] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.448917] LustreError: 347073:0:(osd_handler.c:8532:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5346.458283] LustreError: 347073:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5346.460824] LustreError: 347073:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5346.462890] LustreError: 347073:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5346.465135] LustreError: 347073:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5348.754937] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5348.757602] LustreError: 347471:0:(osd_handler.c:8532:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5348.764933] LustreError: 347471:0:(obd_config.c:843:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5348.767514] LustreError: 347471:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5348.770199] LustreError: 347471:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5348.773345] LustreError: 347471:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5354.862191] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5355.770471] Key type lgssc unregistered [ 5355.895322] LNet: 348456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5355.897631] LNetError: 348456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5355.906280] LNet: Removed LNI 192.168.202.134@tcp [ 5356.166609] Key type .llcrypt unregistered [ 5356.167641] Key type ._llcrypt unregistered [ 5361.810892] Key type ._llcrypt registered [ 5361.811912] Key type .llcrypt registered [ 5361.843866] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 5368.170417] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5368.715649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5368.723722] alg: No test for adler32 (adler32-zlib) [ 5369.569110] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5369.649591] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5371.231130] Key type lgssc registered [ 5371.554734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5382.239147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5382.244050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5383.328763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5383.339784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5383.375573] Lustre: lustre-MDT0000: new disk, initializing [ 5383.398385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.404277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5384.593232] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5388.643609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.689970] Lustre: 351552:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5388.717268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5388.719518] Lustre: Skipped 1 previous similar message [ 5388.758054] Lustre: lustre-MDT0001: new disk, initializing [ 5388.796720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5388.811636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5388.815800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5390.090522] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5393.119366] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5394.270463] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5396.420905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5396.522534] Lustre: lustre-OST0000: new disk, initializing [ 5396.524281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5396.553166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5398.288503] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5398.516654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5398.520905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5398.547219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5401.232797] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5402.556340] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5402.621924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5404.029602] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5404.104901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5408.736572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5408.737840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5408.739253] LustreError: Skipped 1 previous similar message [ 5408.743911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5408.747289] Lustre: Skipped 1 previous similar message [ 5410.899160] Lustre: server umount lustre-OST0000 complete [ 5413.856708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5413.857295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5413.860165] Lustre: Skipped 2 previous similar messages [ 5413.864403] Lustre: Skipped 1 previous similar message [ 5418.264923] Lustre: server umount lustre-MDT0000 complete [ 5419.518126] LustreError: 352329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428636 with bad export cookie 13690708516654073273 [ 5419.521563] LustreError: 352329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5419.521679] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5419.645245] Lustre: server umount lustre-MDT0001 complete [ 5422.244067] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5422.768847] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:10:38 (1777428638) [ 5424.939193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.101483] LustreError: 355605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5425.110440] LustreError: 355605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5425.123766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5426.402436] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5429.257279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5429.376375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5430.725716] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5432.471836] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5433.700797] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5435.983447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5437.965234] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5439.737651] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5441.928699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5441.991176] Lustre: lustre-OST0001: new disk, initializing [ 5441.993463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5442.031263] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5442.034531] Lustre: Skipped 1 previous similar message [ 5443.832622] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5446.940311] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5447.149632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5447.152398] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5447.164831] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5455.335477] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.337403] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5455.339898] Lustre: Skipped 1 previous similar message [ 5457.574098] Lustre: lustre-OST0001: Not available for connect from 192.168.202.34@tcp (stopping) [ 5457.575880] Lustre: Skipped 1 previous similar message [ 5459.197459] Lustre: server umount lustre-OST0001 complete [ 5460.452385] LustreError: 358710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5460.460756] LustreError: 358710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5465.573033] LustreError: 357200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.577261] LustreError: 357200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5470.690881] LustreError: 357199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5470.696010] LustreError: 357199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5475.296318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5475.300318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5475.307258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5475.813659] LustreError: 357200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5475.825096] LustreError: 357200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5477.326367] Lustre: server umount lustre-OST0000 complete [ 5480.929782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.929830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5480.934552] Lustre: Skipped 2 previous similar messages [ 5480.938725] LustreError: Skipped 1 previous similar message [ 5484.759680] Lustre: server umount lustre-MDT0000 complete [ 5485.971224] LustreError: 355587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428702 with bad export cookie 13690708516654074141 [ 5485.974198] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5485.974635] LustreError: 355587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5500.383141] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5500.485187] Lustre: server umount lustre-MDT0001 complete [ 5504.241918] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5505.238860] Key type lgssc unregistered [ 5505.377514] LNet: 359965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5505.379725] LNetError: 359965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5505.387424] LNet: Removed LNI 192.168.202.134@tcp [ 5505.690827] Key type .llcrypt unregistered [ 5505.692051] Key type ._llcrypt unregistered [ 5524.233415] Key type ._llcrypt registered [ 5524.234438] Key type .llcrypt registered [ 5524.281067] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5524.752966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5524.783966] alg: No test for adler32 (adler32-zlib) [ 5525.644243] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5525.726736] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5527.311180] Key type lgssc registered [ 5527.722986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5530.406633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5530.412921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.498400] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5531.505188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5531.562358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5532.767799] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5535.632729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.663688] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5535.676260] Lustre: 362722:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5535.771968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5537.027361] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5538.827197] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5539.989784] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5540.833158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5540.833665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5540.834040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5540.838315] Lustre: Skipped 1 previous similar message [ 5545.952516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5545.954657] Lustre: Skipped 2 previous similar messages [ 5546.836578] Lustre: server umount lustre-MDT0000 complete [ 5548.101593] LustreError: 362714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428764 with bad export cookie 15828032103686674990 [ 5548.104177] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5548.105120] LustreError: 362714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5548.224899] Lustre: server umount lustre-MDT0001 complete [ 5551.424548] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:12:47 (1777428767) [ 5553.677312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.859619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5555.196740] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5557.929795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.055685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5559.469688] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5561.114102] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5562.173491] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5564.425580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5564.494973] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5564.561737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5566.383463] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5568.184691] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5577.316500] Lustre: Permanently deactivating lustre-OST0000 [ 5577.321434] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5579.565532] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5580.477287] LustreError: 367543:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5580.480116] LustreError: 367543:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5580.482097] LustreError: 367543:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5581.075878] LustreError: 367591:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5581.078078] LustreError: 367591:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5587.793840] Lustre: server umount lustre-OST0000 complete [ 5588.821046] LustreError: 367840:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5588.823684] LustreError: 367840:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5593.057097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.058573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5593.062499] Lustre: Skipped 1 previous similar message [ 5598.175986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5598.177576] Lustre: Skipped 2 previous similar messages [ 5600.249268] Lustre: server umount lustre-MDT0000 complete [ 5601.446354] LustreError: 364978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428818 with bad export cookie 15828032103686675487 [ 5601.449392] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5601.449875] LustreError: 364978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5612.050795] Lustre: server umount lustre-MDT0001 complete [ 5614.885295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.134069] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5617.056942] LustreError: 368982:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5619.119549] Lustre: 369321:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.134@tcp [ 5619.981633] Lustre: server umount MGS complete [ 5623.828556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.982828] LustreError: 370043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5623.987401] LustreError: 370043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5623.987575] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5623.991204] Lustre: Skipped 1 previous similar message [ 5624.012531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5625.271627] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5628.038445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.125499] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5629.440264] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5631.690717] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5632.906448] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5633.541272] Lustre: Permanently reactivating lustre-OST0000 [ 5633.546463] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5633.548363] Lustre: Skipped 2 previous similar messages [ 5663.711136] LustreError: 371771:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5666.438656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5666.568184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5666.569985] Lustre: Skipped 1 previous similar message [ 5668.428153] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5670.228340] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5683.168014] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5683.170189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5683.173638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5684.193954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5688.141799] Lustre: server umount lustre-OST0000 complete [ 5689.823959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5689.826060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5689.829065] Lustre: Skipped 1 previous similar message [ 5694.432105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5694.432672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5694.434460] Lustre: Skipped 3 previous similar messages [ 5695.513360] Lustre: server umount lustre-MDT0000 complete [ 5696.757291] LustreError: 370024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428913 with bad export cookie 15828032103686676509 [ 5696.759828] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5696.760775] LustreError: 370024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5696.886738] Lustre: server umount lustre-MDT0001 complete [ 5700.425887] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5701.489251] Key type lgssc unregistered [ 5701.634420] LNet: 374506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5701.637214] LNetError: 374506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5701.644451] LNet: Removed LNI 192.168.202.134@tcp [ 5701.973936] Key type .llcrypt unregistered [ 5701.976121] Key type ._llcrypt unregistered [ 5708.187154] Key type ._llcrypt registered [ 5708.188121] Key type .llcrypt registered [ 5708.224639] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 5714.312680] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5714.616360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5714.626300] alg: No test for adler32 (adler32-zlib) [ 5715.474353] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5715.566140] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5717.151212] Key type lgssc registered [ 5717.498195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5729.259381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5729.268938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5730.391938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5730.403477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5730.450765] Lustre: lustre-MDT0000: new disk, initializing [ 5730.476709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5730.483601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5732.032538] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5736.692545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5736.754622] Lustre: 377617:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5736.797908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5736.802526] Lustre: Skipped 1 previous similar message [ 5736.855158] Lustre: lustre-MDT0001: new disk, initializing [ 5736.896057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5736.911528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5736.917832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5738.455642] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5741.649315] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5742.991248] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5745.403704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5745.512810] Lustre: lustre-OST0000: new disk, initializing [ 5745.514815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5745.544162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.539214] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5750.631119] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5751.931586] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5753.851054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5753.854228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5753.884321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5754.048082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5755.358044] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5755.424136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5758.944781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5758.946908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5758.948708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5758.951030] Lustre: Skipped 1 previous similar message [ 5762.206530] Lustre: server umount lustre-OST0000 complete [ 5764.064507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5764.065292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5764.068282] Lustre: Skipped 1 previous similar message [ 5764.069785] Lustre: Skipped 1 previous similar message [ 5769.184226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5769.185890] Lustre: Skipped 2 previous similar messages [ 5769.631581] Lustre: server umount lustre-MDT0000 complete [ 5770.773737] LustreError: 377609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428987 with bad export cookie 8033853091656757283 [ 5770.777587] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5770.779788] LustreError: 377609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5770.903777] Lustre: server umount lustre-MDT0001 complete [ 5773.462287] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5773.940863] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:16:29 (1777428989) [ 5776.058575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5776.239107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5777.538106] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5780.261626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.386685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5781.595496] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5783.290081] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5784.493737] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5787.497172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5789.677492] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5791.415502] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5792.063182] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5802.725642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5805.024076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5805.026274] LustreError: Skipped 1 previous similar message [ 5805.027891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5805.032361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5806.049819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5810.318780] Lustre: server umount lustre-OST0000 complete [ 5816.288844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5816.288852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5816.289484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5816.289489] Lustre: Skipped 2 previous similar messages [ 5816.292327] Lustre: Skipped 2 previous similar messages [ 5817.686667] Lustre: server umount lustre-MDT0000 complete [ 5819.123841] LustreError: 381707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777429035 with bad export cookie 8033853091656758151 [ 5819.127985] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5819.128355] LustreError: 381707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5819.251528] Lustre: server umount lustre-MDT0001 complete [ 5823.023262] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5824.224790] Key type lgssc unregistered [ 5824.349501] LNet: 385071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5824.353253] LNetError: 385071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5824.360381] LNet: Removed LNI 192.168.202.134@tcp [ 5824.662471] Key type .llcrypt unregistered [ 5824.663608] Key type ._llcrypt unregistered [ 5832.147181] Key type ._llcrypt registered [ 5832.148262] Key type .llcrypt registered [ 5832.192640] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5832.296405] Key type .llcrypt unregistered [ 5832.297554] Key type ._llcrypt unregistered [ 5839.010194] Key type ._llcrypt registered [ 5839.011186] Key type .llcrypt registered [ 5839.045467] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5839.349130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5839.390084] alg: No test for adler32 (adler32-zlib) [ 5840.260277] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5840.355235] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5841.943126] Key type lgssc registered [ 5842.317284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5845.029992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5845.039557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5846.211110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5847.535784] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5850.522197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5850.647181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5852.554631] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5854.378872] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5855.712519] LustreError: 387224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.717450] LustreError: 387224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5856.578491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5856.709064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5857.766881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5857.984468] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5861.856447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5861.858845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5861.862137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5862.881323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5862.881601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.883630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5866.897088] Lustre: server umount lustre-OST0000 complete [ 5872.095911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5872.097329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5872.099709] LustreError: Skipped 1 previous similar message [ 5872.104371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5872.106890] Lustre: Skipped 1 previous similar message [ 5874.272950] Lustre: server umount lustre-MDT0000 complete [ 5875.514174] LustreError: 387203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777429092 with bad export cookie 314349423835064573 [ 5875.516043] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5875.519504] LustreError: 387203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5875.640271] Lustre: server umount lustre-MDT0001 complete [ 5879.051464] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5880.035713] Key type lgssc unregistered [ 5880.154346] LNet: 390208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5880.156410] LNetError: 390208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5880.165294] LNet: Removed LNI 192.168.202.134@tcp [ 5880.459370] Key type .llcrypt unregistered [ 5880.460322] Key type ._llcrypt unregistered [ 5889.027172] Key type ._llcrypt registered [ 5889.028169] Key type .llcrypt registered [ 5889.065106] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5889.424829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5889.468951] alg: No test for adler32 (adler32-zlib) [ 5890.318203] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5890.396333] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5891.975158] Key type lgssc registered [ 5892.326021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5894.790178] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5894.795755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5910.239385] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5915.359171] LustreError: 391558:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 5941.983241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5943.734455] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5945.571429] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5948.327061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5948.507346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5948.534293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5949.956436] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5952.936992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5953.057631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5954.313017] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5956.280761] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5957.693617] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5962.211429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.215042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5962.215317] Lustre: Skipped 1 previous similar message [ 5967.316913] Lustre: server umount lustre-OST0000 complete [ 5971.937317] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5971.939611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.943318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5971.944915] Lustre: Skipped 1 previous similar message [ 5974.879421] Lustre: server umount lustre-MDT0000 complete [ 5976.108146] LustreError: 391566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777429192 with bad export cookie 2723844945449852448 [ 5976.111284] LustreError: 391566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5976.111868] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5976.223696] Lustre: server umount lustre-MDT0001 complete [ 5980.020628] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5981.035805] Key type lgssc unregistered [ 5981.160651] LNet: 394770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5981.164364] LNetError: 394770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5981.173562] LNet: Removed LNI 192.168.202.134@tcp [ 5981.474549] Key type .llcrypt unregistered [ 5981.476374] Key type ._llcrypt unregistered [ 5990.341883] Key type ._llcrypt registered [ 5990.342899] Key type .llcrypt registered [ 5990.378480] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5990.688823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5990.697352] alg: No test for adler32 (adler32-zlib) [ 5991.553019] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 5991.640335] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5993.223253] Key type lgssc registered [ 5993.636165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5996.345197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5996.350939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.503280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.792703] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6002.045450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.199174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.430314] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6005.390137] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6006.517180] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6009.130487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6009.251053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.157334] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6012.906156] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6022.566103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6027.747529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6027.750001] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6027.750520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6027.752771] Lustre: Skipped 2 previous similar messages [ 6029.476102] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 6029.478452] Lustre: Skipped 2 previous similar messages [ 6031.447790] Lustre: server umount lustre-MDT0000 complete [ 6032.138283] LustreError: 396127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.34@tcp arrived at 1777429248 with bad export cookie 15928135807467811775 [ 6032.142794] LustreError: 396127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6032.838063] LustreError: 396128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777429249 with bad export cookie 15928135807467811124 [ 6032.841932] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6032.863976] LustreError: 396146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6032.864669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.864690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6032.867900] LustreError: 396146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6032.871657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6037.984406] LustreError: 396869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6037.985346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6037.993180] LustreError: 396869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6038.930160] Lustre: server umount lustre-OST0000 complete [ 6041.269711] Lustre: server umount lustre-MDT0001 complete [ 6045.111250] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 6046.117631] Key type lgssc unregistered [ 6046.235494] LNet: 399500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6046.239185] LNetError: 399500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6046.247342] LNet: Removed LNI 192.168.202.134@tcp [ 6046.559234] Key type .llcrypt unregistered [ 6046.560331] Key type ._llcrypt unregistered [ 6055.802749] Key type ._llcrypt registered [ 6055.804626] Key type .llcrypt registered [ 6055.850739] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6056.211474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6056.248560] alg: No test for adler32 (adler32-zlib) [ 6057.102502] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 6057.193000] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 6058.775147] Key type lgssc registered [ 6059.142680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6061.741878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6061.749580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.955668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6064.302506] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6067.076872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6067.221877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6068.729625] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6070.615337] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6071.826466] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6074.162201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6074.319475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6076.261584] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6078.191193] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6087.659085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6092.772073] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6092.772562] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6092.776059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6092.781659] Lustre: Skipped 1 previous similar message [ 6093.989259] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [ 6095.826513] Lustre: server umount lustre-MDT0001 complete [ 6097.888382] LustreError: 400875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6097.888816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6097.889121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.889130] Lustre: Skipped 1 previous similar message [ 6097.889599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6097.892805] LustreError: 400875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6103.008330] LustreError: 402174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.008976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6103.506105] Lustre: server umount lustre-OST0000 complete [ 6104.817727] Lustre: server umount lustre-MDT0000 complete [ 6109.108429] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 6110.015337] Key type lgssc unregistered [ 6110.127461] LNet: 404229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6110.131098] LNetError: 404229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6110.137294] LNet: Removed LNI 192.168.202.134@tcp [ 6110.401792] Key type .llcrypt unregistered [ 6110.402884] Key type ._llcrypt unregistered [ 6118.204413] Key type ._llcrypt registered [ 6118.205479] Key type .llcrypt registered [ 6118.242200] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 6124.220904] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6124.576186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6124.585707] alg: No test for adler32 (adler32-zlib) [ 6125.436120] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 6125.519412] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 6127.103179] Key type lgssc registered [ 6127.499153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6139.059262] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6144.542347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6144.547307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.642319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6145.654376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6145.691191] Lustre: lustre-MDT0000: new disk, initializing [ 6145.734094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6145.740500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6147.003116] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6152.700850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6152.806764] Lustre: lustre-OST0000: new disk, initializing [ 6152.808846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6152.810505] Lustre: Skipped 1 previous similar message [ 6152.836598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6153.478467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6153.483408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6153.497375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6154.794768] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6157.913810] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6163.403137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.443947] Lustre: 408164:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6163.471756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6163.515709] Lustre: lustre-MDT0001: new disk, initializing [ 6163.558500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6163.570780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6163.573725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6164.772105] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6168.203217] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6168.270782] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6169.489706] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6169.554192] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6174.178742] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6174.178765] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.184772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6174.186459] Lustre: Skipped 1 previous similar message [ 6175.395679] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [ 6177.171619] Lustre: server umount lustre-MDT0001 complete [ 6184.417958] LustreError: 408171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.422072] LustreError: 408171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6189.537355] LustreError: 408174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.537896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6189.538060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6189.550463] Lustre: Skipped 1 previous similar message [ 6189.552862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6194.657181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6194.657609] LustreError: 408175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.531630] Lustre: server umount lustre-OST0000 complete [ 6203.075946] Lustre: server umount lustre-MDT0000 complete [ 6207.629964] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 6208.747374] Key type lgssc unregistered [ 6208.874497] LNet: 412730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6208.878274] LNetError: 412730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6208.889457] LNet: Removed LNI 192.168.202.134@tcp [ 6209.203983] Key type .llcrypt unregistered [ 6209.205615] Key type ._llcrypt unregistered [ 6221.837185] Key type ._llcrypt registered [ 6221.838208] Key type .llcrypt registered [ 6221.880674] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 6221.979199] Key type .llcrypt unregistered [ 6221.980215] Key type ._llcrypt unregistered [ 6228.548880] Key type ._llcrypt registered [ 6228.550057] Key type .llcrypt registered [ 6228.592592] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6229.052501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6229.133292] alg: No test for adler32 (adler32-zlib) [ 6229.995692] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 6230.092331] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 6231.679247] Key type lgssc registered [ 6232.094569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6240.726695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6240.735176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.842314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6241.854537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6241.892682] Lustre: lustre-MDT0000: new disk, initializing [ 6241.917378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6241.923710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6243.189763] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6247.210948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6247.255992] Lustre: 416045:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6247.283876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6247.286234] Lustre: Skipped 1 previous similar message [ 6247.328500] Lustre: lustre-MDT0001: new disk, initializing [ 6247.364377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6247.376088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6247.379796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6248.608792] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6251.690114] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6252.852331] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6255.102706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6255.233075] Lustre: lustre-OST0000: new disk, initializing [ 6255.235310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6255.266140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6256.255884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6256.259561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6256.302470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6257.175835] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6260.313555] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6262.751828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6262.754198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.758091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6266.849579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6266.850386] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.765149] Lustre: server umount lustre-MDT0000 complete [ 6269.980602] LustreError: 416035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777429486 with bad export cookie 9064290382384337918 [ 6269.984424] LustreError: 416035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6269.985071] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6270.105228] Lustre: server umount lustre-MDT0001 complete [ 6281.499296] Lustre: server umount lustre-OST0000 complete [ 6284.662168] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:25:00 (1777429500) [ 6287.175950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6287.372400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6288.620675] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6291.372083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.495994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6292.745674] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6294.371238] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6295.481238] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6310.625869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6310.695781] Lustre: Found index 0 for lustre-OST0000, updating log [ 6310.752435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6312.437618] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6312.804976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6313.988064] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6317.089104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6317.089479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6317.094336] Lustre: Skipped 1 previous similar message [ 6317.095877] Lustre: Skipped 2 previous similar messages [ 6321.097650] Lustre: server umount lustre-OST0000 complete [ 6334.432465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.433156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6334.435178] Lustre: Skipped 1 previous similar message [ 6334.438533] Lustre: Skipped 3 previous similar messages [ 6339.551967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6339.553645] Lustre: Skipped 2 previous similar messages [ 6339.665500] Lustre: server umount lustre-MDT0000 complete [ 6340.829099] LustreError: 419679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777429557 with bad export cookie 9064290382384339297 [ 6340.832046] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6340.832352] LustreError: 419679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6340.940969] Lustre: server umount lustre-MDT0001 complete [ 6343.664660] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:25:59 (1777429559) [ 6379.615249] LNet: 1 local NIs in recovery (showing 1): 192.168.202.134@tcp [ 6924.195172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.425745] LustreError: 425767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.433670] LustreError: 425767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6924.453876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6926.164585] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6929.553501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6931.376136] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6933.564470] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6935.396589] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6951.780734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6951.910917] Lustre: Found index 0 for lustre-OST0000, updating log [ 6953.066668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6954.446459] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6957.082699] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6958.560375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6958.567437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.579751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6959.650422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6959.654176] Lustre: Skipped 1 previous similar message [ 6964.131627] Lustre: server umount lustre-OST0000 complete [ 6965.535527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.536393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6965.540536] Lustre: Skipped 2 previous similar messages [ 6969.632816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6969.637958] Lustre: Skipped 8 previous similar messages [ 6971.681905] LustreError: 425786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6971.688764] LustreError: 425786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6971.702387] Lustre: server umount lustre-MDT0000 complete [ 6972.705259] LustreError: 426974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.714883] LustreError: 426974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6973.480831] LustreError: 425750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777430190 with bad export cookie 9064290382384340354 [ 6973.487574] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6973.488391] LustreError: 425750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6973.619176] Lustre: server umount lustre-MDT0001 complete [ 6980.703357] LNet: 1 local NIs in recovery (showing 1): 192.168.202.134@tcp [ 6990.847432] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 6993.863309] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7005.696912] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:37:01 (1777430221) [ 7011.513131] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7012.690607] Key type lgssc unregistered [ 7013.855537] LNet: 432040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7013.857790] LNetError: 432040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7083.493581] LNet: Removed LNI 192.168.202.134@tcp [ 7083.875383] Key type .llcrypt unregistered [ 7083.877055] Key type ._llcrypt unregistered [ 7091.841117] Key type ._llcrypt registered [ 7091.842111] Key type .llcrypt registered [ 7091.875239] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7092.328612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7092.338546] alg: No test for adler32 (adler32-zlib) [ 7093.209198] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 7093.308202] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7094.903182] Key type lgssc registered [ 7095.494517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7096.956148] LNet: Added LNI 192.168.202.134@tcp100 [8/256/0/180] [ 7097.249857] LNet: Added LNI 192.168.202.134@tcp101 [8/256/0/180] [ 7097.643621] LNet: Added LNI 192.168.202.134@tcp102 [8/256/0/180] [ 7098.068122] LNet: Added LNI 192.168.202.134@tcp103 [8/256/0/180] [ 7098.504325] LNet: Added LNI 192.168.202.134@tcp104 [8/256/0/180] [ 7098.981244] LNet: Added LNI 192.168.202.134@tcp105 [8/256/0/180] [ 7099.465679] LNet: Added LNI 192.168.202.134@tcp106 [8/256/0/180] [ 7099.946239] LNet: Added LNI 192.168.202.134@tcp107 [8/256/0/180] [ 7100.420529] LNet: Added LNI 192.168.202.134@tcp108 [8/256/0/180] [ 7100.747022] LNet: Added LNI 192.168.202.134@tcp109 [8/256/0/180] [ 7101.037160] LNet: Added LNI 192.168.202.134@tcp110 [8/256/0/180] [ 7101.318844] LNet: Added LNI 192.168.202.134@tcp111 [8/256/0/180] [ 7101.612682] LNet: Added LNI 192.168.202.134@tcp112 [8/256/0/180] [ 7101.911337] LNet: Added LNI 192.168.202.134@tcp113 [8/256/0/180] [ 7102.253576] LNet: Added LNI 192.168.202.134@tcp114 [8/256/0/180] [ 7102.662344] LNet: Added LNI 192.168.202.134@tcp115 [8/256/0/180] [ 7103.063904] LNet: Added LNI 192.168.202.134@tcp116 [8/256/0/180] [ 7103.473840] LNet: Added LNI 192.168.202.134@tcp117 [8/256/0/180] [ 7103.876321] LNet: Added LNI 192.168.202.134@tcp118 [8/256/0/180] [ 7104.326595] LNet: Added LNI 192.168.202.134@tcp119 [8/256/0/180] [ 7104.760471] LNet: Added LNI 192.168.202.134@tcp120 [8/256/0/180] [ 7105.207828] LNet: Added LNI 192.168.202.134@tcp121 [8/256/0/180] [ 7105.650626] LNet: Added LNI 192.168.202.134@tcp122 [8/256/0/180] [ 7105.936970] LNet: Added LNI 192.168.202.134@tcp123 [8/256/0/180] [ 7106.214946] LNet: Added LNI 192.168.202.134@tcp124 [8/256/0/180] [ 7106.494451] LNet: Added LNI 192.168.202.134@tcp125 [8/256/0/180] [ 7106.787195] LNet: Added LNI 192.168.202.134@tcp126 [8/256/0/180] [ 7107.070144] LNet: Added LNI 192.168.202.134@tcp127 [8/256/0/180] [ 7107.352897] LNet: Added LNI 192.168.202.134@tcp128 [8/256/0/180] [ 7107.639508] LNet: Added LNI 192.168.202.134@tcp129 [8/256/0/180] [ 7107.948739] LNet: Added LNI 192.168.202.134@tcp130 [8/256/0/180] [ 7108.272431] LNet: Added LNI 192.168.202.134@tcp131 [8/256/0/180] [ 7108.550817] LNet: Added LNI 192.168.202.134@tcp132 [8/256/0/180] [ 7108.852874] LNet: Added LNI 192.168.202.134@tcp133 [8/256/0/180] [ 7109.268424] LNet: Added LNI 192.168.202.134@tcp134 [8/256/0/180] [ 7109.730821] LNet: Added LNI 192.168.202.134@tcp135 [8/256/0/180] [ 7112.403648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7112.408631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7113.451282] Lustre: 435020:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.134@tcp: can't add NID 192.168.202.134@tcp130: rc = -75 [ 7113.515452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7113.530157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7113.575359] Lustre: lustre-MDT0000: new disk, initializing [ 7113.605654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.613050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7115.075075] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7120.501828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7120.547731] Lustre: 435037:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7120.569332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7120.571055] Lustre: Skipped 1 previous similar message [ 7120.607874] Lustre: lustre-MDT0001: new disk, initializing [ 7120.618640] Lustre: 435817:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.134@tcp: can't add NID 192.168.202.134@tcp130: rc = -75 [ 7120.621196] Lustre: 435817:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7120.658636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7120.670744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7120.674190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7121.871657] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7123.394197] Lustre: 436164:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.134@tcp: can't add NID 192.168.202.134@tcp130: rc = -75 [ 7123.401288] Lustre: 436164:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7125.329978] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7126.645487] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7129.653870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7129.828468] Lustre: lustre-OST0000: new disk, initializing [ 7129.831526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7129.886075] Lustre: 436784:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.134@tcp: can't add NID 192.168.202.134@tcp130: rc = -75 [ 7129.890322] Lustre: 436784:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7129.914715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.148193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7131.151279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7131.193349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7132.285188] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7136.001928] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7141.351267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7141.351387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.356707] LustreError: Skipped 1 previous similar message [ 7141.357669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7141.368055] Lustre: Skipped 1 previous similar message [ 7141.378396] Lustre: Skipped 1 previous similar message [ 7145.191502] Lustre: server umount lustre-OST0000 complete [ 7151.586079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7151.586286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.587176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7151.601461] Lustre: Skipped 1 previous similar message [ 7156.705234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7156.710499] Lustre: Skipped 2 previous similar messages [ 7161.311118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7161.402706] Lustre: server umount lustre-MDT0000 complete [ 7163.147574] LustreError: 435029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777430379 with bad export cookie 1687048305550043209 [ 7163.153451] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7163.156280] LustreError: 435029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7163.312487] Lustre: server umount lustre-MDT0001 complete [ 7168.133555] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7169.534435] Key type lgssc unregistered [ 7169.692810] LNet: 438566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7169.697932] LNetError: 438566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7169.703958] LNet: Removed LNI 192.168.202.134@tcp [ 7170.719162] LNet: Removed LNI 192.168.202.134@tcp100 [ 7170.723238] LNetError: 438566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7170.730187] LNetError: 438566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7170.737681] LNet: Removed LNI 192.168.202.134@tcp101 [ 7171.743149] LNet: Removed LNI 192.168.202.134@tcp102 [ 7171.746886] LNetError: 438566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7171.752024] LNetError: 438566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7171.757374] LNet: Removed LNI 192.168.202.134@tcp103 [ 7171.760577] LNet: Removed LNI 192.168.202.134@tcp104 [ 7171.763652] LNet: Removed LNI 192.168.202.134@tcp105 [ 7171.766861] LNet: Removed LNI 192.168.202.134@tcp106 [ 7171.770087] LNet: Removed LNI 192.168.202.134@tcp107 [ 7171.772924] LNet: Removed LNI 192.168.202.134@tcp108 [ 7171.775635] LNet: Removed LNI 192.168.202.134@tcp109 [ 7171.778213] LNet: Removed LNI 192.168.202.134@tcp110 [ 7171.781310] LNet: Removed LNI 192.168.202.134@tcp111 [ 7171.784519] LNet: Removed LNI 192.168.202.134@tcp112 [ 7171.787765] LNet: Removed LNI 192.168.202.134@tcp113 [ 7171.790966] LNet: Removed LNI 192.168.202.134@tcp114 [ 7171.793638] LNet: Removed LNI 192.168.202.134@tcp115 [ 7171.796395] LNet: Removed LNI 192.168.202.134@tcp116 [ 7171.799130] LNet: Removed LNI 192.168.202.134@tcp117 [ 7171.802449] LNet: Removed LNI 192.168.202.134@tcp118 [ 7171.805663] LNet: Removed LNI 192.168.202.134@tcp119 [ 7171.808911] LNet: Removed LNI 192.168.202.134@tcp120 [ 7171.811500] LNet: Removed LNI 192.168.202.134@tcp121 [ 7171.814082] LNet: Removed LNI 192.168.202.134@tcp122 [ 7171.816482] LNet: Removed LNI 192.168.202.134@tcp123 [ 7171.819050] LNet: Removed LNI 192.168.202.134@tcp124 [ 7171.822098] LNet: Removed LNI 192.168.202.134@tcp125 [ 7171.824964] LNet: Removed LNI 192.168.202.134@tcp126 [ 7171.827584] LNet: Removed LNI 192.168.202.134@tcp127 [ 7171.830194] LNet: Removed LNI 192.168.202.134@tcp128 [ 7171.832425] LNet: Removed LNI 192.168.202.134@tcp129 [ 7171.834723] LNet: Removed LNI 192.168.202.134@tcp130 [ 7171.836854] LNet: Removed LNI 192.168.202.134@tcp131 [ 7171.839121] LNet: Removed LNI 192.168.202.134@tcp132 [ 7171.841842] LNet: Removed LNI 192.168.202.134@tcp133 [ 7171.844192] LNet: Removed LNI 192.168.202.134@tcp134 [ 7171.852430] LNet: Removed LNI 192.168.202.134@tcp135 [ 7172.232677] Key type .llcrypt unregistered [ 7172.235891] Key type ._llcrypt unregistered [ 7182.398967] Key type ._llcrypt registered [ 7182.400843] Key type .llcrypt registered [ 7182.465981] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 7189.923445] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7190.233460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7190.243441] alg: No test for adler32 (adler32-zlib) [ 7191.096471] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 7191.180603] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7192.767141] Key type lgssc registered [ 7193.244422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7208.962280] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:40:24 (1777430424) [ 7217.707036] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7219.309578] Key type lgssc unregistered [ 7219.485656] LNet: 442571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7219.490400] LNetError: 442571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7219.501773] LNet: Removed LNI 192.168.202.134@tcp [ 7219.954767] Key type .llcrypt unregistered [ 7219.956638] Key type ._llcrypt unregistered [ 7228.355033] Key type ._llcrypt registered [ 7228.356161] Key type .llcrypt registered [ 7228.401966] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7228.859975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7228.895077] alg: No test for adler32 (adler32-zlib) [ 7229.785995] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 7229.916517] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7231.503169] Key type lgssc registered [ 7232.184286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7237.067391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7237.076844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7238.192814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7238.209053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7238.264010] Lustre: lustre-MDT0000: new disk, initializing [ 7238.294637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.303444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7240.045702] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7245.503978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.539239] Lustre: 443742:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7245.559274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7245.560977] Lustre: Skipped 1 previous similar message [ 7245.595862] Lustre: lustre-MDT0001: new disk, initializing [ 7245.624210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7245.634024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7245.636635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7247.060524] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7250.636149] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7251.859033] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7255.022177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7255.182422] Lustre: lustre-OST0000: new disk, initializing [ 7255.185668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7255.225471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7257.796139] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7261.754803] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7263.120039] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing lnet_if_list [ 7264.764086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7264.768678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7264.805348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7265.231929] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7265.248532] LNet: Added LNI 192.168.202.134@tcp10 [8/256/0/180] [ 7267.618112] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7267.634348] LNet: Added LNI 192.168.202.134@tcp11 [8/256/0/180] [ 7269.445719] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7269.462314] LNet: Added LNI 192.168.202.134@tcp12 [8/256/0/180] [ 7271.208375] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7271.223120] LNet: Added LNI 192.168.202.134@tcp13 [8/256/0/180] [ 7272.917965] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7272.941112] LNet: Added LNI 192.168.202.134@tcp14 [8/256/0/180] [ 7274.705282] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7274.718313] LNet: Added LNI 192.168.202.134@tcp15 [8/256/0/180] [ 7276.562299] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7276.583294] LNet: Added LNI 192.168.202.134@tcp16 [8/256/0/180] [ 7278.547487] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7278.564301] LNet: Added LNI 192.168.202.134@tcp17 [8/256/0/180] [ 7280.681355] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7280.698695] LNet: Added LNI 192.168.202.134@tcp18 [8/256/0/180] [ 7282.797865] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7282.812627] LNet: Added LNI 192.168.202.134@tcp19 [8/256/0/180] [ 7284.745620] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7284.762420] LNet: Added LNI 192.168.202.134@tcp20 [8/256/0/180] [ 7286.936794] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7286.955782] LNet: Added LNI 192.168.202.134@tcp21 [8/256/0/180] [ 7288.850147] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7288.861744] LNet: Added LNI 192.168.202.134@tcp22 [8/256/0/180] [ 7291.185343] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7291.204561] LNet: Added LNI 192.168.202.134@tcp23 [8/256/0/180] [ 7293.659162] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7293.668558] LNet: Added LNI 192.168.202.134@tcp24 [8/256/0/180] [ 7295.801599] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7295.821108] LNet: Added LNI 192.168.202.134@tcp25 [8/256/0/180] [ 7298.207280] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7298.219623] LNet: Added LNI 192.168.202.134@tcp26 [8/256/0/180] [ 7300.688118] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7300.700973] LNet: Added LNI 192.168.202.134@tcp27 [8/256/0/180] [ 7303.241248] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7303.256983] LNet: Added LNI 192.168.202.134@tcp28 [8/256/0/180] [ 7305.628349] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7305.641153] LNet: Added LNI 192.168.202.134@tcp29 [8/256/0/180] [ 7307.809604] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7307.820965] LNet: Added LNI 192.168.202.134@tcp30 [8/256/0/180] [ 7310.131897] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7310.146847] LNet: Added LNI 192.168.202.134@tcp31 [8/256/0/180] [ 7312.705586] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7312.727200] LNet: Added LNI 192.168.202.134@tcp32 [8/256/0/180] [ 7315.379364] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7315.394798] LNet: Added LNI 192.168.202.134@tcp33 [8/256/0/180] [ 7318.008919] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7318.022168] LNet: Added LNI 192.168.202.134@tcp34 [8/256/0/180] [ 7320.766566] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7320.783615] LNet: Added LNI 192.168.202.134@tcp35 [8/256/0/180] [ 7323.331671] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7323.349208] LNet: Added LNI 192.168.202.134@tcp36 [8/256/0/180] [ 7326.528452] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7326.552619] LNet: Added LNI 192.168.202.134@tcp37 [8/256/0/180] [ 7329.440699] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7329.452044] LNet: Added LNI 192.168.202.134@tcp38 [8/256/0/180] [ 7332.364952] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7332.379685] LNet: Added LNI 192.168.202.134@tcp39 [8/256/0/180] [ 7335.264354] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7335.275089] LNet: Added LNI 192.168.202.134@tcp40 [8/256/0/180] [ 7337.610974] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7337.621404] LNet: Added LNI 192.168.202.134@tcp41 [8/256/0/180] [ 7340.708670] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7340.718994] LNet: Added LNI 192.168.202.134@tcp42 [8/256/0/180] [ 7343.873718] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7343.890992] LNet: Added LNI 192.168.202.134@tcp43 [8/256/0/180] [ 7347.313439] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7347.331381] LNet: Added LNI 192.168.202.134@tcp44 [8/256/0/180] [ 7350.053924] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7350.071306] LNet: Added LNI 192.168.202.134@tcp45 [8/256/0/180] [ 7353.332524] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7353.352739] LNet: Added LNI 192.168.202.134@tcp46 [8/256/0/180] [ 7356.891196] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7356.912900] LNet: Added LNI 192.168.202.134@tcp47 [8/256/0/180] [ 7360.210459] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7360.218882] LNet: Added LNI 192.168.202.134@tcp48 [8/256/0/180] [ 7363.056337] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7363.077282] LNet: Added LNI 192.168.202.134@tcp49 [8/256/0/180] [ 7366.406281] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7366.424187] LNet: Added LNI 192.168.202.134@tcp50 [8/256/0/180] [ 7377.378092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.381265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.385850] Lustre: Skipped 2 previous similar messages [ 7377.389947] Lustre: Skipped 1 previous similar message [ 7380.502623] Lustre: server umount lustre-MDT0000 complete [ 7381.575497] LustreError: 453694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777430598 with bad export cookie 3319739309548821692 [ 7381.579391] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7381.582345] LustreError: 453694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7381.726555] Lustre: server umount lustre-MDT0001 complete [ 7389.275226] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7392.203197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7398.815087] Lustre: 443020:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777430599/real 1777430599] req@00000000174601e5 x1863770907434496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777430615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7398.820983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.911136] Lustre: 443020:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777430603/real 1777430603] req@000000005af851b5 x1863770907434624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777430619 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7403.999075] Lustre: 443017:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777430604/real 1777430604] req@00000000adb21109 x1863770907434752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777430620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7407.008529] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2e121550348ee4d4 [ 7407.013975] Lustre: MGC192.168.202.134@tcp: Connection restored to 0@lo (at 0@lo) [ 7407.192811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7408.223122] Lustre: 443017:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777430608/real 1777430608] req@00000000af554d60 x1863770907435008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777430624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7408.693930] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7411.585081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7411.739813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7412.703164] Lustre: 443020:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777430613/real 1777430613] req@00000000317d38d1 x1863770907435136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777430629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7413.328111] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7415.150407] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7416.665140] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7416.805312] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7417.827441] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7427.041714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7427.046439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7427.054529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7427.058140] Lustre: Skipped 1 previous similar message [ 7427.551864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7428.064463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7428.068409] Lustre: Skipped 2 previous similar messages [ 7431.243280] Lustre: server umount lustre-OST0000 complete [ 7433.186264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.194449] Lustre: Skipped 1 previous similar message [ 7433.198475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7433.202096] Lustre: Skipped 1 previous similar message [ 7438.304971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.308713] Lustre: Skipped 1 previous similar message [ 7439.084957] Lustre: server umount lustre-MDT0000 complete [ 7440.286047] LustreError: 444522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777430656 with bad export cookie 3319739309548823764 [ 7440.288835] LustreError: 444522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7440.289026] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7440.393804] Lustre: server umount lustre-MDT0001 complete [ 7444.878011] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7445.740149] Key type lgssc unregistered [ 7445.885541] LNet: 471423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7445.887883] LNetError: 471423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7445.889650] LNet: Removed LNI 192.168.202.134@tcp [ 7445.890734] LNet: Removed LNI 192.168.202.134@tcp10 [ 7445.891754] LNet: Removed LNI 192.168.202.134@tcp11 [ 7445.892734] LNet: Removed LNI 192.168.202.134@tcp12 [ 7445.893727] LNet: Removed LNI 192.168.202.134@tcp13 [ 7445.894724] LNet: Removed LNI 192.168.202.134@tcp14 [ 7445.895707] LNet: Removed LNI 192.168.202.134@tcp15 [ 7445.896650] LNet: Removed LNI 192.168.202.134@tcp16 [ 7445.897596] LNet: Removed LNI 192.168.202.134@tcp17 [ 7445.898579] LNet: Removed LNI 192.168.202.134@tcp18 [ 7445.899559] LNet: Removed LNI 192.168.202.134@tcp19 [ 7445.900545] LNet: Removed LNI 192.168.202.134@tcp20 [ 7445.901525] LNet: Removed LNI 192.168.202.134@tcp21 [ 7445.902447] LNet: Removed LNI 192.168.202.134@tcp22 [ 7445.903407] LNet: Removed LNI 192.168.202.134@tcp23 [ 7445.904410] LNet: Removed LNI 192.168.202.134@tcp24 [ 7445.905356] LNet: Removed LNI 192.168.202.134@tcp25 [ 7445.906333] LNet: Removed LNI 192.168.202.134@tcp26 [ 7445.907250] LNet: Removed LNI 192.168.202.134@tcp27 [ 7445.908230] LNet: Removed LNI 192.168.202.134@tcp28 [ 7445.909188] LNet: Removed LNI 192.168.202.134@tcp29 [ 7445.910180] LNet: Removed LNI 192.168.202.134@tcp30 [ 7445.911191] LNet: Removed LNI 192.168.202.134@tcp31 [ 7445.912151] LNet: Removed LNI 192.168.202.134@tcp32 [ 7445.913113] LNet: Removed LNI 192.168.202.134@tcp33 [ 7445.914053] LNet: Removed LNI 192.168.202.134@tcp34 [ 7445.914911] LNet: Removed LNI 192.168.202.134@tcp35 [ 7445.915883] LNet: Removed LNI 192.168.202.134@tcp36 [ 7445.916847] LNet: Removed LNI 192.168.202.134@tcp37 [ 7445.917859] LNet: Removed LNI 192.168.202.134@tcp38 [ 7445.918779] LNet: Removed LNI 192.168.202.134@tcp39 [ 7445.919757] LNet: Removed LNI 192.168.202.134@tcp40 [ 7445.920673] LNet: Removed LNI 192.168.202.134@tcp41 [ 7445.921624] LNet: Removed LNI 192.168.202.134@tcp42 [ 7445.922555] LNet: Removed LNI 192.168.202.134@tcp43 [ 7445.923493] LNet: Removed LNI 192.168.202.134@tcp44 [ 7445.924402] LNet: Removed LNI 192.168.202.134@tcp45 [ 7445.925308] LNet: Removed LNI 192.168.202.134@tcp46 [ 7445.926234] LNet: Removed LNI 192.168.202.134@tcp47 [ 7445.927198] LNet: Removed LNI 192.168.202.134@tcp48 [ 7445.928136] LNet: Removed LNI 192.168.202.134@tcp49 [ 7445.936289] LNet: Removed LNI 192.168.202.134@tcp50 [ 7446.155696] Key type .llcrypt unregistered [ 7446.156598] Key type ._llcrypt unregistered [ 7455.424170] Key type ._llcrypt registered [ 7455.425139] Key type .llcrypt registered [ 7455.456063] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7455.749860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7455.793807] alg: No test for adler32 (adler32-zlib) [ 7456.656193] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 7456.733328] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7458.311102] Key type lgssc registered [ 7458.888822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7461.621680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7461.627350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7462.809567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7464.458336] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7467.356058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7467.535399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7469.282538] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7471.173328] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7472.421074] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7475.564144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7475.665860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7477.381209] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7480.080891] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7489.004279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7495.570244] loop0: detected capacity change from 0 to 204800000 [ 7495.623090] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7495.626132] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7495.629812] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7498.095200] loop0: detected capacity change from 0 to 204800000 [ 7498.166713] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7498.170627] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7498.174738] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7500.473535] loop0: detected capacity change from 0 to 204800000 [ 7502.901187] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7502.943078] Lustre: fs15246-OST0000: new disk, initializing [ 7502.944633] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7502.959884] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7504.648184] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7509.503158] loop1: detected capacity change from 0 to 204800000 [ 7512.057857] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7512.112892] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7512.142908] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7512.195638] Lustre: fs15246-MDT0000: new disk, initializing [ 7512.260690] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7512.271042] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7512.333068] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7513.376420] Lustre: 477028:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000950755c1 x1863771145405952/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777430745 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7513.820427] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7518.434203] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7518.440667] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7518.450648] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7527.912320] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7527.920593] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7530.351591] Lustre: server umount fs15246-MDT0000 complete [ 7533.030718] LustreError: 472669:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.041190] LustreError: 472669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7534.712102] Lustre: server umount fs15246-OST0000 complete [ 7540.192182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7540.195165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7540.201664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7543.268495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.268829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7543.280375] Lustre: Skipped 1 previous similar message [ 7545.499744] Lustre: server umount lustre-OST0000 complete [ 7548.384956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7548.385898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7548.388297] Lustre: Skipped 1 previous similar message [ 7553.324467] Lustre: server umount lustre-MDT0000 complete [ 7553.504325] LustreError: 472645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7553.512399] LustreError: 472645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7555.112221] LustreError: 472626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777430771 with bad export cookie 13198248308984300821 [ 7555.116713] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7555.122545] LustreError: 472626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7555.277780] Lustre: server umount lustre-MDT0001 complete [ 7559.846987] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7561.349150] Key type lgssc unregistered [ 7561.524662] LNet: 480091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7561.529360] LNetError: 480091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7561.541638] LNet: Removed LNI 192.168.202.134@tcp [ 7561.994972] Key type .llcrypt unregistered [ 7561.997044] Key type ._llcrypt unregistered [ 7570.700252] Key type ._llcrypt registered [ 7570.701353] Key type .llcrypt registered [ 7570.740558] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7571.093200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7571.125862] alg: No test for adler32 (adler32-zlib) [ 7571.985155] Lustre: Lustre: Build Version: 2.17.51_75_gf89bdc1 [ 7572.064920] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7573.639201] Key type lgssc registered [ 7574.142914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7591.581498] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 7595.542993] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7611.713269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7611.725504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.842062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7612.860122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7612.913221] Lustre: lustre-MDT0000: new disk, initializing [ 7612.945805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7612.956095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7614.518075] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7619.988301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.061447] Lustre: 484477:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7620.101415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7620.105015] Lustre: Skipped 1 previous similar message [ 7620.157422] Lustre: lustre-MDT0001: new disk, initializing [ 7620.194318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7620.210846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7620.215062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7621.882605] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7625.852131] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7627.485462] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7630.658142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7630.754038] Lustre: lustre-OST0000: new disk, initializing [ 7630.756098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7630.783113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7633.248571] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7633.544840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7633.551198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7633.604896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7637.209704] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7638.971581] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7639.070767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7640.920542] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7641.039837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7644.131726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.131799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7644.135413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7644.139066] Lustre: Skipped 1 previous similar message [ 7644.143706] LustreError: Skipped 1 previous similar message [ 7648.086309] Lustre: server umount lustre-OST0000 complete [ 7650.784439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7650.789050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.797267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7650.800909] Lustre: Skipped 1 previous similar message [ 7654.368883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7654.369922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7659.488792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7659.492511] Lustre: Skipped 2 previous similar messages [ 7663.585223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7663.588834] Lustre: Skipped 1 previous similar message [ 7664.095116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7664.188678] Lustre: server umount lustre-MDT0000 complete [ 7665.995963] LustreError: 485257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777430882 with bad export cookie 5847397883824370945 [ 7666.001547] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7666.002879] LustreError: 485257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7666.147894] Lustre: server umount lustre-MDT0001 complete [ 7675.690225] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 7679.931209] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7690.914525] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7442 sec ========== 22:48:26 (1777430906) [ 7691.662235] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:48:27 (1777430907) === [ 7693.077185] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:48:28 (1777430908) === [ 7711.313792] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7712.480315] Key type lgssc unregistered [ 7712.619591] LNet: 492653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7712.623125] LNetError: 492653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7712.630420] LNet: Removed LNI 192.168.202.134@tcp [ 7712.986255] Key type .llcrypt unregistered [ 7712.987765] Key type ._llcrypt unregistered