[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 614547445 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002411] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.006752] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009213] LSM: Security Framework initializing [ 0.010131] Yama: becoming mindful. [ 0.011060] SELinux: Initializing. [ 0.012076] *** VALIDATE selinux *** [ 0.018588] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022773] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024013] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026096] *** VALIDATE tmpfs *** [ 0.028150] *** VALIDATE proc *** [ 0.029209] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.031252] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035031] Spectre V2 : User space: Vulnerable [ 0.036141] Speculative Store Bypass: Vulnerable [ 0.041682] debug: unmapping init [mem 0xffffffff9fe59000-0xffffffff9fe60fff] [ 0.044013] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046334] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051029] ... max period: 00007fffffffffff [ 0.052011] ... fixed-purpose events: 3 [ 0.053017] ... event mask: 000000070000000f [ 0.054340] rcu: Hierarchical SRCU implementation. [ 0.056738] smp: Bringing up secondary CPUs ... [ 0.057725] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.071183] smp: Brought up 1 node, 4 CPUs [ 0.073032] smpboot: Max logical packages: 1 [ 0.074022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.114000] node 0 deferred pages initialised in 33ms [ 0.117008] devtmpfs: initialized [ 0.119313] x86/mm: Memory block size: 128MB [ 0.125170] gcov: version magic: 0x41383552 [ 0.129941] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.130110] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.132239] pinctrl core: initialized pinctrl subsystem [ 0.133186] [ 0.133724] ************************************************************* [ 0.134019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135021] ** ** [ 0.136018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.137021] ** ** [ 0.138021] ** This means that this kernel is built to expose internal ** [ 0.139016] ** IOMMU data structures, which may compromise security on ** [ 0.140022] ** your system. ** [ 0.141019] ** ** [ 0.142020] ** If you see this message and you are not debugging the ** [ 0.143021] ** kernel, report this immediately to your vendor! ** [ 0.144018] ** ** [ 0.145019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146023] ************************************************************* [ 0.147933] NET: Registered protocol family 16 [ 0.148560] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.149088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.150081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.152202] cpuidle: using governor menu [ 0.154000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.158090] PCI: Using configuration type 1 for base access [ 0.160184] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.171115] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.174130] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.179137] cryptd: max_cpu_qlen set to 1000 [ 0.187033] ACPI: Added _OSI(Module Device) [ 0.190021] ACPI: Added _OSI(Processor Device) [ 0.193020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201952] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209932] ACPI: Interpreter enabled [ 0.210088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211018] ACPI: Using IOAPIC for interrupt routing [ 0.212120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213644] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226831] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.227067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.229097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.231255] acpiphp: Slot [2] registered [ 0.232141] acpiphp: Slot [3] registered [ 0.233071] acpiphp: Slot [4] registered [ 0.234095] acpiphp: Slot [5] registered [ 0.235225] acpiphp: Slot [6] registered [ 0.236193] acpiphp: Slot [7] registered [ 0.237139] acpiphp: Slot [8] registered [ 0.238133] acpiphp: Slot [9] registered [ 0.239159] acpiphp: Slot [10] registered [ 0.240152] acpiphp: Slot [11] registered [ 0.242063] acpiphp: Slot [12] registered [ 0.243128] acpiphp: Slot [13] registered [ 0.244122] acpiphp: Slot [14] registered [ 0.245129] acpiphp: Slot [15] registered [ 0.246148] acpiphp: Slot [16] registered [ 0.247128] acpiphp: Slot [17] registered [ 0.248134] acpiphp: Slot [18] registered [ 0.249087] acpiphp: Slot [19] registered [ 0.250133] acpiphp: Slot [20] registered [ 0.251077] acpiphp: Slot [21] registered [ 0.252092] acpiphp: Slot [22] registered [ 0.253170] acpiphp: Slot [23] registered [ 0.254138] acpiphp: Slot [24] registered [ 0.255080] acpiphp: Slot [25] registered [ 0.256064] acpiphp: Slot [26] registered [ 0.257131] acpiphp: Slot [27] registered [ 0.258091] acpiphp: Slot [28] registered [ 0.259093] acpiphp: Slot [29] registered [ 0.260130] acpiphp: Slot [30] registered [ 0.261107] acpiphp: Slot [31] registered [ 0.262083] PCI host bridge to bus 0000:00 [ 0.263015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.264039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.265029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.266359] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.267031] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.268030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.269161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.273836] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.276756] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.282063] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.285199] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.286016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.287017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.288016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.289572] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.290939] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.291047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.292563] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.295014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.303074] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.305017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.308902] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.312021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.316038] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.323018] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.329412] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.332018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.335017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.342020] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.347692] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.350016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.353016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.364021] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.370554] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.373022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.376018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.383016] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.392024] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.395015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.398024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.405016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.411374] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.414015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.417016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.424019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.431197] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.432535] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.433495] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.434449] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.435313] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.439111] iommu: Default domain type: Passthrough [ 0.440721] SCSI subsystem initialized [ 0.441000] ACPI: bus type USB registered [ 0.444159] usbcore: registered new interface driver usbfs [ 0.447158] usbcore: registered new interface driver hub [ 0.448105] usbcore: registered new device driver usb [ 0.453229] pps_core: LinuxPPS API ver. 1 registered [ 0.457018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.464063] PTP clock support registered [ 0.467032] EDAC MC: Ver: 3.0.0 [ 0.470536] PCI: Using ACPI for IRQ routing [ 0.474000] NetLabel: Initializing [ 0.476013] NetLabel: domain hash size = 128 [ 0.480013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.483082] NetLabel: unlabeled traffic allowed by default [ 0.485300] vgaarb: loaded [ 0.487313] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.488010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.493661] clocksource: Switched to clocksource kvm-clock [ 0.664496] VFS: Disk quotas dquot_6.6.0 [ 0.665784] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.668353] *** VALIDATE ramfs *** [ 0.669462] *** VALIDATE hugetlbfs *** [ 0.670957] pnp: PnP ACPI init [ 0.678427] pnp: PnP ACPI: found 6 devices [ 0.706972] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.710541] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.712857] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.715142] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.717251] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.718782] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.721224] NET: Registered protocol family 2 [ 0.723322] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.730983] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.734901] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.740610] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.744825] TCP: Hash tables configured (established 65536 bind 65536) [ 0.747678] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.750886] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.754062] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.757376] NET: Registered protocol family 1 [ 0.761935] RPC: Registered named UNIX socket transport module. [ 0.764486] RPC: Registered udp transport module. [ 0.766200] RPC: Registered tcp transport module. [ 0.767854] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.772952] NET: Registered protocol family 44 [ 0.774457] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.776319] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.781077] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.783432] PCI: CLS 0 bytes, default 64 [ 0.785983] Unpacking initramfs... [ 3.523761] debug: unmapping init [mem 0xffff8ceb7cc54000-0xffff8ceb7ffbffff] [ 3.535407] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.539617] software IO TLB: mapped [mem 0x00000000a2800000-0x00000000a6800000] (64MB) [ 3.549317] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.316392] Initialise system trusted keyrings [ 4.317958] Key type blacklist registered [ 4.331231] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.362141] zbud: loaded [ 4.373424] *** VALIDATE nfs *** [ 4.374332] *** VALIDATE nfs4 *** [ 4.375527] pstore: using deflate compression [ 4.387212] Platform Keyring initialized [ 4.711542] NET: Registered protocol family 38 [ 4.713251] Key type asymmetric registered [ 4.714777] Asymmetric key parser 'x509' registered [ 4.717287] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.722779] io scheduler mq-deadline registered [ 4.724969] io scheduler kyber registered [ 4.726605] io scheduler bfq registered [ 4.730954] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.734546] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.738123] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.741672] ACPI: Power Button [PWRF] [ 4.870552] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.007311] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.354343] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.581929] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.886886] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.936173] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.989040] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.994762] Non-volatile memory driver v1.3 [ 5.997014] Linux agpgart interface v0.103 [ 6.033062] virtio_blk virtio1: [vda] 133056 512-byte logical blocks (68.1 MB/65.0 MiB) [ 6.042824] vda: detected capacity change from 0 to 68124672 [ 6.066645] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.069890] vdb: detected capacity change from 0 to 1073741824 [ 6.124860] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.131446] vdc: detected capacity change from 0 to 2621440000 [ 6.166177] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.179937] vdd: detected capacity change from 0 to 2621440000 [ 6.219296] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.225326] vde: detected capacity change from 0 to 4294967296 [ 6.258698] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.261020] vdf: detected capacity change from 0 to 4294967296 [ 6.275202] libphy: Fixed MDIO Bus: probed [ 6.280170] usbcore: registered new interface driver usbserial_generic [ 6.281879] usbserial: USB Serial support registered for generic [ 6.283652] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.290558] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.293258] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.296861] mousedev: PS/2 mouse device common for all mice [ 6.301909] rtc_cmos 00:05: RTC can wake from S4 [ 6.309021] rtc_cmos 00:05: registered as rtc0 [ 6.311845] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.313484] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.322869] intel_pstate: CPU model not supported [ 6.325982] hid: raw HID events driver (C) Jiri Kosina [ 6.326186] usbcore: registered new interface driver usbhid [ 6.326190] usbhid: USB HID core driver [ 6.326343] drop_monitor: Initializing network drop monitor service [ 6.326494] Initializing XFRM netlink socket [ 6.326876] NET: Registered protocol family 10 [ 6.338207] Segment Routing with IPv6 [ 6.361200] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.362824] NET: Registered protocol family 17 [ 6.363851] mpls_gso: MPLS GSO support [ 6.387853] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.398362] RAS: Correctable Errors collector initialized. [ 6.401469] AVX version of gcm_enc/dec engaged. [ 6.406550] AES CTR mode by8 optimization enabled [ 6.689840] sched_clock: Marking stable (6689441950, 0)->(8772085665, -2082643715) [ 6.708536] registered taskstats version 1 [ 6.712854] Loading compiled-in X.509 certificates [ 6.714730] zswap: loaded using pool lzo/zbud [ 6.806359] Key type big_key registered [ 6.847756] Key type encrypted registered [ 6.849440] ima: No TPM chip found, activating TPM-bypass! [ 6.851243] ima: Allocated hash algorithm: sha1 [ 6.853966] ima: No architecture policies found [ 6.856167] evm: Initialising EVM extended attributes: [ 6.859359] evm: security.selinux [ 6.860589] evm: security.ima [ 6.861634] evm: security.capability [ 6.863460] evm: HMAC attrs: 0x1 [ 6.867710] rtc_cmos 00:05: setting system clock to 2025-09-02 10:33:56 UTC (1756809236) [ 6.877720] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 6.880621] debug: unmapping init [mem 0xffffffff9fb82000-0xffffffff9fe58fff] [ 6.891458] Write protecting the kernel read-only data: 28672k [ 6.895113] debug: unmapping init [mem 0xffffffff9e203000-0xffffffff9e3fffff] [ 6.902662] debug: unmapping init [mem 0xffffffff9eb14000-0xffffffff9ebfffff] [ 6.950873] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.962424] systemd[1]: Detected virtualization kvm. [ 6.964607] systemd[1]: Detected architecture x86-64. [ 6.966739] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.005225] systemd[1]: No hostname configured. [ 7.006639] systemd[1]: Set hostname to . [ 7.008635] random: systemd: uninitialized urandom read (16 bytes read) [ 7.015637] systemd[1]: Initializing machine ID from random generator. [ 7.189188] random: ln: uninitialized urandom read (6 bytes read) [ 7.463625] random: systemd: uninitialized urandom read (16 bytes read) [ 7.466057] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.474521] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 7.480177] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.825590] device-mapper: uevent: version 1.0.3 [ 9.828338] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook...[ 11.672156] virtio_net virtio0 ens2: renamed from eth0 [ 12.012566] scsi host0: ata_piix [ 12.035943] scsi host1: ata_piix [ 12.037401] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.051662] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.076140] random: fast init done [ 17.393328] random: crng init done [ 17.396362] random: 7 urandom warning(s) missed due to ratelimiting [ 20.614294] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.036248] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.292948] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.354862] SELinux: Disabled at runtime. [ 26.465812] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.477736] systemd[1]: Detected virtualization kvm. [ 26.481167] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.570812] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.578958] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.598712] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.619106] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.633933] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.656960] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.720586] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ 29.246824] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 31.411240] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.658191] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.819285] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.394823] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.484925] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 40.098288] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 40.893849] NFS: Registering the id_resolver key type [ 40.895977] Key type id_resolver registered [ 40.899433] Key type id_legacy registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg221-server login: [ 77.069687] libcfs: loading out-of-tree module taints kernel. [ 77.080375] Key type ._llcrypt registered [ 77.082109] Key type .llcrypt registered [ 77.132551] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 86.242717] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 86.882961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.890694] alg: No test for adler32 (adler32-zlib) [ 87.878346] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 88.201229] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 89.823243] Key type lgssc registered [ 90.510683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.394756] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 98.550889] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.113511] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.766812] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.925387] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.599401] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 126.096968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 126.134310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 126.150598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 127.280762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 127.304458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 127.346702] Lustre: lustre-MDT0000: new disk, initializing [ 127.389303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 127.399072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 129.066019] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.978188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.027094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 136.075394] Lustre: 6492:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 136.099212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 136.102742] Lustre: Skipped 1 previous similar message [ 136.152735] Lustre: lustre-MDT0001: new disk, initializing [ 136.194174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 136.212278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 136.217947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 137.895280] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 140.482853] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 144.863879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.895983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 145.023916] Lustre: lustre-OST0000: new disk, initializing [ 145.026757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 145.052559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 145.846705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 145.850785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 145.886920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 147.454423] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.636036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.661342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 153.705709] Lustre: lustre-OST0001: new disk, initializing [ 153.708536] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 153.733742] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 155.203835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 155.207928] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 155.250571] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 156.411245] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.795157] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 168.975861] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 176.714479] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing check_logdir /tmp/testlogs/ [ 179.215552] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing yml_node [ 181.898912] Lustre: DEBUG MARKER: Client: 2.16.58.1 [ 183.180567] Lustre: DEBUG MARKER: MDS: 2.16.58.1 [ 184.646494] Lustre: DEBUG MARKER: OSS: 2.16.58.1 [ 185.558628] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 2 06:36:54 EDT 2025 [ 196.045313] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 197.082387] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 211.425291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 211.427306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 211.434416] Lustre: Skipped 3 previous similar messages [ 211.437023] Lustre: Skipped 2 previous similar messages [ 216.338440] LustreError: 12198:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 216.385194] Lustre: server umount lustre-MDT0000 complete [ 218.634820] LustreError: 7434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809448 with bad export cookie 991937042427974343 [ 218.636594] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 218.640203] LustreError: 7434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 221.664218] LustreError: 10436:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 221.664635] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 221.664930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 221.664935] Lustre: Skipped 1 previous similar message [ 221.671368] LustreError: 10436:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 6 previous similar messages [ 221.676773] Lustre: Skipped 1 previous similar message [ 223.712710] LustreError: 9078:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 223.712780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 223.722737] LustreError: 9078:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 223.728914] Lustre: Skipped 2 previous similar messages [ 224.800181] LustreError: 12400:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 224.804794] LustreError: 12400:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 224.907336] Lustre: server umount lustre-MDT0001 complete [ 227.062115] LustreError: 12601:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 227.064483] LustreError: 12601:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 227.101937] Lustre: server umount lustre-OST0000 complete [ 229.298233] LustreError: 12801:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 229.306750] LustreError: 12801:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 229.404911] Lustre: server umount lustre-OST0001 complete [ 233.265841] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 239.264944] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 246.290079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 247.790021] hrtimer: interrupt took 3011056 ns [ 251.062591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 255.455622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 259.992984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 264.902061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 264.947994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 265.096226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 265.117369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 265.160981] Lustre: lustre-MDT0000: new disk, initializing [ 265.225632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 265.248626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 267.895267] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 275.944551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 275.989690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 276.050185] Lustre: 14982:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 276.104436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 276.107365] Lustre: Skipped 1 previous similar message [ 276.151229] Lustre: lustre-MDT0001: new disk, initializing [ 276.225654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.257225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 276.261937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 278.880955] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 283.713781] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 285.768187] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 290.019626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 290.089070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 291.749440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 291.848421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 294.172795] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 299.197544] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 301.650250] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 301.778612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 304.056260] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 304.189418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 307.175052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.176643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 307.175737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 307.178245] LustreError: Skipped 1 previous similar message [ 307.182214] Lustre: Skipped 1 previous similar message [ 311.344191] LustreError: 17920:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 311.348242] LustreError: 17920:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 311.438683] Lustre: server umount lustre-OST0000 complete [ 317.409152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.409163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 317.410751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.410757] Lustre: Skipped 1 previous similar message [ 317.415360] Lustre: Skipped 1 previous similar message [ 319.374474] LustreError: 18121:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 319.377378] LustreError: 18121:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 319.395888] LustreError: 18121:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (ffff8cebfbf0d000) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 319.437237] Lustre: server umount lustre-MDT0000 complete [ 321.507096] LustreError: 16739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809551 with bad export cookie 991937042427975876 [ 321.507462] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.512540] LustreError: 16739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 321.775981] Lustre: server umount lustre-MDT0001 complete [ 323.849685] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:39:12 (1756809552) [ 325.179677] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 326.106696] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:39:15 (1756809555) [ 329.996192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 330.263682] LustreError: 18993:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.311687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 330.314536] Lustre: Skipped 1 previous similar message [ 332.550482] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 335.329656] LustreError: 18993:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.076835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 339.244774] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 341.737992] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 343.479312] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 347.073581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 350.325362] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 353.064813] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 367.073062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 367.077609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.079657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 367.085964] Lustre: Skipped 1 previous similar message [ 367.088707] Lustre: Skipped 1 previous similar message [ 367.789093] LustreError: 21262:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 367.792494] LustreError: 21262:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 367.876668] Lustre: server umount lustre-OST0000 complete [ 369.573871] Lustre: server umount lustre-MDT0000 complete [ 371.174673] LustreError: 19715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809600 with bad export cookie 991937042427976758 [ 371.179786] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 371.184131] LustreError: 19715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 371.360421] Lustre: server umount lustre-MDT0001 complete [ 376.131515] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 377.628329] Key type lgssc unregistered [ 377.803228] LNet: 22196:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.807416] LNetError: 22196:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 377.818842] LNet: Removed LNI 192.168.202.121@tcp [ 378.269262] Key type .llcrypt unregistered [ 378.270981] Key type ._llcrypt unregistered [ 388.598971] Key type ._llcrypt registered [ 388.600204] Key type .llcrypt registered [ 388.652912] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 389.206302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 389.235303] alg: No test for adler32 (adler32-zlib) [ 390.132728] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 390.251720] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 391.855191] Key type lgssc registered [ 392.438341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 396.315508] loop: module loaded [ 396.337904] loop0: detected capacity change from 0 to 204800000 [ 396.479128] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 396.485573] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 396.626791] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 396.632428] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 396.662655] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 396.690087] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 396.711261] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 396.740751] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 399.418050] loop0: detected capacity change from 0 to 204800000 [ 399.490505] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 399.498527] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 399.505138] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 399.584427] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 401.893417] loop0: detected capacity change from 0 to 204800000 [ 404.726553] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 404.761888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 404.777378] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.903321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 405.923091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 405.964173] Lustre: lustre-MDT0000: new disk, initializing [ 405.999384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.008785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 407.781519] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 412.558183] loop1: detected capacity change from 0 to 204800000 [ 415.798868] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 415.832548] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 415.982380] Lustre: lustre-OST07c6: new disk, initializing [ 415.985852] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 415.989894] Lustre: Skipped 1 previous similar message [ 416.027154] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 419.319760] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 420.911194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 420.924227] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 420.945996] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 502.753586] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 502.754258] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.762905] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 507.759129] LustreError: 26517:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 507.841507] Lustre: server umount lustre-OST07c6 complete [ 512.548763] LustreError: 27018:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 512.551909] LustreError: 27018:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 512.686729] Lustre: server umount lustre-MDT0000 complete [ 519.714918] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 521.160443] Key type lgssc unregistered [ 521.335932] LNet: 27849:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.339662] LNetError: 27849:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 522.407151] LNet: Removed LNI 192.168.202.121@tcp [ 522.809852] Key type .llcrypt unregistered [ 522.811632] Key type ._llcrypt unregistered [ 531.478474] Key type ._llcrypt registered [ 531.479796] Key type .llcrypt registered [ 531.564235] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 532.120301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 532.163533] alg: No test for adler32 (adler32-zlib) [ 533.140265] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 533.271521] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 534.879231] Key type lgssc registered [ 535.475801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 539.578031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 539.589757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.803858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.481060] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 546.548938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 546.724297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 548.655237] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 551.435563] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 553.340121] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 556.641629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 556.808853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.428807] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 562.723114] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 566.059340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 571.230603] LustreError: 31277:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 576.486695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.487326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 576.490473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 576.493286] Lustre: Skipped 1 previous similar message [ 576.502410] LustreError: Skipped 1 previous similar message [ 577.329226] LustreError: 31528:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 577.422427] Lustre: server umount lustre-OST0000 complete [ 581.603324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.609316] Lustre: Skipped 1 previous similar message [ 581.611631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 581.614476] Lustre: Skipped 2 previous similar messages [ 585.554671] LustreError: 31729:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 585.557808] LustreError: 31729:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 585.584591] Lustre: server umount lustre-MDT0000 complete [ 586.721943] LustreError: 29794:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.729322] LustreError: 29794:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 587.427285] LustreError: 29062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809817 with bad export cookie 3570769249397068175 [ 587.432397] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.434229] LustreError: 29062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 587.541823] LustreError: 31930:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 587.545206] LustreError: 31930:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 587.669537] Lustre: server umount lustre-MDT0001 complete [ 592.613677] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 594.036973] Key type lgssc unregistered [ 594.218085] LNet: 32461:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.221732] LNetError: 32461:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.235553] LNet: Removed LNI 192.168.202.121@tcp [ 594.675591] Key type .llcrypt unregistered [ 594.677810] Key type ._llcrypt unregistered [ 603.401485] Key type ._llcrypt registered [ 603.403103] Key type .llcrypt registered [ 603.458033] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 603.944430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 603.953753] alg: No test for adler32 (adler32-zlib) [ 604.837858] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 604.950457] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 606.551237] Key type lgssc registered [ 607.079514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 611.477502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 611.491154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.763498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.726473] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 619.513147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 619.726598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 621.755386] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 624.364261] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 625.977630] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 629.274333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 629.461979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 632.207182] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 634.641257] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 637.673460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 646.112430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 646.115973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.122792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 648.163357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.166261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 648.166767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 648.176057] Lustre: Skipped 1 previous similar message [ 651.246400] LustreError: 35947:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 651.317360] Lustre: server umount lustre-OST0000 complete [ 653.282815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.284161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 653.289725] Lustre: Skipped 1 previous similar message [ 658.402328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 658.406367] Lustre: Skipped 2 previous similar messages [ 659.145891] LustreError: 36149:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 659.147976] LustreError: 36149:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 659.186509] Lustre: server umount lustre-MDT0000 complete [ 660.809091] LustreError: 33685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809890 with bad export cookie 5152437262136277445 [ 660.811197] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.815241] LustreError: 33685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 660.892024] LustreError: 36352:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 660.894092] LustreError: 36352:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 660.998959] Lustre: server umount lustre-MDT0001 complete [ 665.326533] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 666.703845] Key type lgssc unregistered [ 666.860506] LNet: 36884:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.864302] LNetError: 36884:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.872610] LNet: Removed LNI 192.168.202.121@tcp [ 667.256206] Key type .llcrypt unregistered [ 667.257895] Key type ._llcrypt unregistered [ 677.472153] Key type ._llcrypt registered [ 677.473790] Key type .llcrypt registered [ 677.523813] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 677.987128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 677.994428] alg: No test for adler32 (adler32-zlib) [ 678.885096] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 679.004918] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 680.599226] Key type lgssc registered [ 681.100181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 684.486055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 684.499436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.722178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 687.330662] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 690.895816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 691.082533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 692.805988] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 694.965746] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 696.380698] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 699.447990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 699.646399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.193031] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 704.632491] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 710.954584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 720.205209] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 721.377045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 721.380172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 721.381668] Lustre: Skipped 1 previous similar message [ 721.386365] Lustre: Skipped 2 previous similar messages [ 723.910750] LustreError: 40546:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 723.955573] Lustre: server umount lustre-MDT0000 complete [ 725.313878] LustreError: 38273:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 725.322474] LustreError: 38273:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 725.660465] LustreError: 38258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809955 with bad export cookie 7674635206223179938 [ 725.663512] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.666029] LustreError: 38258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 726.497477] LustreError: 38273:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.501541] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 726.514465] Lustre: Skipped 1 previous similar message [ 726.517237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 730.426830] LustreError: 38273:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 730.438269] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) [ 731.792593] LustreError: 40749:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 731.795253] LustreError: 40749:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 731.919162] Lustre: server umount lustre-MDT0001 complete [ 750.318359] LustreError: 40971:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 750.321089] LustreError: 40971:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 750.409751] Lustre: server umount lustre-OST0000 complete [ 757.399625] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 758.810948] Key type lgssc unregistered [ 758.983742] LNet: 41806:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.987819] LNetError: 41806:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.997603] LNet: Removed LNI 192.168.202.121@tcp [ 759.389993] Key type .llcrypt unregistered [ 759.391882] Key type ._llcrypt unregistered [ 770.092236] Key type ._llcrypt registered [ 770.093431] Key type .llcrypt registered [ 770.148265] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 770.554402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 770.567429] alg: No test for adler32 (adler32-zlib) [ 771.464328] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 771.580838] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 773.191279] Key type lgssc registered [ 773.690902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 777.074354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 777.088212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.319506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.942321] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 783.590438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 783.763130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 785.417958] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 787.760399] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 789.347213] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 792.271634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 792.441921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.942886] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 797.226664] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 802.664142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 807.907913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.912760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 807.916139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 807.921208] Lustre: Skipped 1 previous similar message [ 808.786819] Lustre: lustre-OST0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 812.294892] LustreError: 45427:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 812.527925] Lustre: server umount lustre-OST0000 complete [ 833.506254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.509617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 833.519240] Lustre: Skipped 2 previous similar messages [ 836.530562] LustreError: 45805:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 836.543858] LustreError: 45805:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 836.727679] Lustre: server umount lustre-MDT0000 complete [ 838.624026] LustreError: 43182:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 838.665533] LustreError: 43182:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 841.338419] LustreError: 43162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810070 with bad export cookie 15470818920877205688 [ 841.356421] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 847.733831] LustreError: 46008:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 847.749473] LustreError: 46008:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 848.215530] Lustre: server umount lustre-MDT0001 complete [ 862.273456] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 866.338913] Key type lgssc unregistered [ 866.633527] LNet: 46540:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.636883] LNetError: 46540:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.687825] LNet: Removed LNI 192.168.202.121@tcp [ 868.582949] Key type .llcrypt unregistered [ 868.585985] Key type ._llcrypt unregistered [ 886.214679] Key type ._llcrypt registered [ 886.216566] Key type .llcrypt registered [ 886.316756] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 887.064491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 887.084945] alg: No test for adler32 (adler32-zlib) [ 888.037294] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 888.178376] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 889.793471] Key type lgssc registered [ 890.392133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 895.237210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 895.261598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.539656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.976316] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 903.896175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 904.146784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 906.598377] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 910.090495] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 912.168341] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 916.325271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 916.558920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.236464] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 921.708858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 923.653357] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 932.448396] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 934.933667] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 946.505262] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-09-02 6h49m34s [ 947.413360] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 949.730144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.731383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.741229] Lustre: Skipped 2 previous similar messages [ 949.743278] Lustre: Skipped 1 previous similar message [ 950.818500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 950.821532] Lustre: Skipped 3 previous similar messages [ 952.703627] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 952.709343] Lustre: Skipped 1 previous similar message [ 954.891672] LustreError: 50481:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 954.936223] Lustre: server umount lustre-MDT0000 complete [ 955.875632] LustreError: 47913:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 955.885357] LustreError: 47913:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 956.905522] LustreError: 50482:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.920554] LustreError: 50482:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 957.016897] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 960.856704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.951802] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.109181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.161729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 962.150756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 963.642492] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 965.853075] Lustre: DEBUG MARKER: Wait for df (35949) ... [ 966.858602] Lustre: DEBUG MARKER: done [ 970.211027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 970.215233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.224636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 971.235660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 971.240133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.160116] LustreError: 51530:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 975.172338] LustreError: 51530:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 975.310872] Lustre: server umount lustre-OST0000 complete [ 977.888934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.889037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 977.889848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 977.889853] Lustre: Skipped 3 previous similar messages [ 977.900488] Lustre: Skipped 1 previous similar message [ 983.429708] LustreError: 51736:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 983.432586] LustreError: 51736:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 983.464561] Lustre: server umount lustre-MDT0000 complete [ 983.968498] LustreError: 47918:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 983.976579] LustreError: 47918:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 985.025166] LustreError: 47899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810214 with bad export cookie 851242997968620874 [ 985.029180] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 985.033717] LustreError: 47899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 985.199089] Lustre: server umount lustre-MDT0001 complete [ 989.913955] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 991.282460] Key type lgssc unregistered [ 991.480495] LNet: 52468:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.485638] LNetError: 52468:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.500200] LNet: Removed LNI 192.168.202.121@tcp [ 991.916809] Key type .llcrypt unregistered [ 991.918961] Key type ._llcrypt unregistered [ 1012.439238] Key type ._llcrypt registered [ 1012.440851] Key type .llcrypt registered [ 1012.502375] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 1021.787621] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1022.467785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.478199] alg: No test for adler32 (adler32-zlib) [ 1023.411664] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1023.564697] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1025.183182] Key type lgssc registered [ 1025.768657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1030.746081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1034.950074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1039.063319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1043.319157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1047.668462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1047.712712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1047.734923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.952996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1048.975878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1049.025680] Lustre: lustre-MDT0000: new disk, initializing [ 1049.104314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.125934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1051.608674] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1059.558909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1059.617951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1059.695716] Lustre: 56395:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1059.740869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1059.746871] Lustre: Skipped 1 previous similar message [ 1059.850771] Lustre: lustre-MDT0001: new disk, initializing [ 1059.934483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1059.968077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1059.978020] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1062.683649] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1067.927206] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1070.616575] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1075.323466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1075.379912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1075.565862] Lustre: lustre-OST0000: new disk, initializing [ 1075.571740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1075.653761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.381628] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1084.487196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1084.494107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1084.586195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1085.058634] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1087.978859] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1088.145515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1091.127558] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1091.288310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1095.136408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1095.137995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.141633] LustreError: Skipped 1 previous similar message [ 1095.142687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.149394] Lustre: Skipped 1 previous similar message [ 1095.160486] Lustre: Skipped 1 previous similar message [ 1098.741153] LustreError: 59332:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1098.833943] Lustre: server umount lustre-OST0000 complete [ 1105.377230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.384445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1105.387449] Lustre: Skipped 1 previous similar message [ 1107.287625] LustreError: 59534:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1107.302048] LustreError: 59534:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1107.464272] Lustre: server umount lustre-MDT0000 complete [ 1109.816481] LustreError: 56388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810339 with bad export cookie 13363427553603634626 [ 1109.821473] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1109.823384] LustreError: 56388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1109.916783] LustreError: 59735:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1109.923923] LustreError: 59735:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1110.079219] Lustre: server umount lustre-MDT0001 complete [ 1115.955976] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 06:52:24 (1756810344) [ 1120.769046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.106355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.013266] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1126.371648] LustreError: 60406:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.395053] LustreError: 60406:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1130.467119] LustreError: 60406:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1131.048526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1131.280219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1133.935507] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1137.925365] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1141.005609] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1145.990343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1146.311105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1150.520548] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1154.212109] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1173.809939] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-09-02 6h53m22s [ 1174.813921] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1179.827402] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1181.872897] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1182.059708] LustreError: 61981:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1202.524520] Lustre: lustre-MDT0000: Client a4af4e5d-272f-431c-9482-7d3114549deb (at 192.168.202.21@tcp) reconnecting [ 1222.121729] LustreError: 61981:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1223.575642] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1225.229893] Lustre: DEBUG MARKER: done [ 1232.355542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.355977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1232.367292] Lustre: Skipped 1 previous similar message [ 1232.380245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.383142] Lustre: Skipped 1 previous similar message [ 1234.486426] LustreError: 63433:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1234.489410] LustreError: 63433:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1234.628669] Lustre: server umount lustre-OST0000 complete [ 1237.475401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.478416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.481769] Lustre: Skipped 2 previous similar messages [ 1243.553381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1243.555915] Lustre: Skipped 3 previous similar messages [ 1251.295163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1251.378363] LustreError: 63634:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1251.381445] LustreError: 63634:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1251.479262] Lustre: server umount lustre-MDT0000 complete [ 1253.867429] LustreError: 60406:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.374583] LustreError: 60386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810484 with bad export cookie 13363427553603635515 [ 1254.382439] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.387606] LustreError: 60386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1254.668945] Lustre: server umount lustre-MDT0001 complete [ 1261.586977] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1263.408670] Key type lgssc unregistered [ 1263.623715] LNet: 64367:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.631669] LNetError: 64367:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.642847] LNet: Removed LNI 192.168.202.121@tcp [ 1264.159915] Key type .llcrypt unregistered [ 1264.161637] Key type ._llcrypt unregistered [ 1284.098151] Key type ._llcrypt registered [ 1284.099833] Key type .llcrypt registered [ 1284.172697] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 1293.409789] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1293.982088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1294.001959] alg: No test for adler32 (adler32-zlib) [ 1294.960668] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1295.100155] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1296.727173] Key type lgssc registered [ 1297.270981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1302.843953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1306.383547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1309.672168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1313.266483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1316.759148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1316.791389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1316.801300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.925518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1317.946733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1317.989928] Lustre: lustre-MDT0000: new disk, initializing [ 1318.035084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.044959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1320.159361] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1325.841340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1325.876668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1325.923927] Lustre: 68293:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1325.963533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1325.966251] Lustre: Skipped 1 previous similar message [ 1326.015834] Lustre: lustre-MDT0001: new disk, initializing [ 1326.081833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1326.104391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1326.109769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1328.151192] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1332.494667] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1334.429510] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1337.583929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1337.620393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1337.741789] Lustre: lustre-OST0000: new disk, initializing [ 1337.744770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1337.782577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.379313] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1344.252653] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1346.055310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1346.060749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1346.118366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1346.294820] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1346.402964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1348.441870] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1348.545427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1351.136633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1351.139557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.146538] LustreError: Skipped 1 previous similar message [ 1351.147367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1351.161186] Lustre: Skipped 1 previous similar message [ 1355.502149] LustreError: 71232:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1355.591526] Lustre: server umount lustre-OST0000 complete [ 1361.376805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.377939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1361.382126] Lustre: Skipped 1 previous similar message [ 1361.386451] Lustre: Skipped 2 previous similar messages [ 1363.774786] LustreError: 71433:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1363.777423] LustreError: 71433:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1363.811145] Lustre: server umount lustre-MDT0000 complete [ 1365.398080] LustreError: 69074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810595 with bad export cookie 16303905360691146807 [ 1365.398948] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1365.404538] LustreError: 69074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1365.480838] LustreError: 71635:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1365.482956] LustreError: 71635:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1365.591438] Lustre: server umount lustre-MDT0001 complete [ 1369.307028] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 06:56:38 (1756810598) [ 1370.160737] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1371.132923] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 06:56:39 (1756810599) [ 1377.907611] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1380.216353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.523921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.159864] Lustre: Failing over lustre-MDT0000 [ 1381.168277] LustreError: 72711:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1381.173544] LustreError: 72711:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1381.184709] LustreError: 72711:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1381.277119] LustreError: 72762:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1381.280517] LustreError: 72762:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1381.408209] Lustre: server umount lustre-MDT0000 complete [ 1385.148651] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:56:53 (1756810613) [ 1389.204868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.466972] LustreError: 73330:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.478371] LustreError: 73330:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1389.522111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.680123] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1394.659236] LustreError: 73330:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.231476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1398.455585] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1401.677024] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1403.825591] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1408.209914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1408.445403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.448909] Lustre: Skipped 1 previous similar message [ 1412.372961] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1415.766279] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1423.226913] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1427.424841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1427.433125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.446739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.191582] LustreError: 75673:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1429.193608] LustreError: 75673:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1429.227157] Lustre: server umount lustre-MDT0000 complete [ 1431.556308] LustreError: 73311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810661 with bad export cookie 16303905360691147948 [ 1431.561597] LustreError: 73311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1431.562953] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1431.781150] Lustre: server umount lustre-MDT0001 complete [ 1433.931729] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1436.784834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1442.031413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1447.391255] Lustre: 66508:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810661/real 1756810661] req@0000000005da2a61 x1842149352075008/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756810677 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1447.402393] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.999178] Lustre: 66506:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810665/real 1756810665] req@00000000643ba969 x1842149352075392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756810681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1452.021707] Lustre: 66506:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1456.543231] Lustre: 66508:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810670/real 1756810670] req@00000000f67f9cb3 x1842149352075648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756810686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1456.561543] Lustre: 66508:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1456.608414] LustreError: 66504:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000060304a23 x1842149352076544/t0(0) o250->MGC192.168.202.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1456.810363] LustreError: 74937:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.871507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.938345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1457.894757] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1459.458841] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1463.007285] Lustre: 66506:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810676/real 1756810676] req@00000000ae30aff8 x1842149352075904/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756810692 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1463.020646] Lustre: 66506:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1463.266858] LustreError: 74936:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.286928] LustreError: 74936:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1464.422515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1466.605813] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1469.208816] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1469.938424] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1471.114785] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1475.040903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1475.043144] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.046285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.050235] Lustre: Skipped 2 previous similar messages [ 1480.163610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1480.166864] Lustre: Skipped 4 previous similar messages [ 1480.275347] LustreError: 78135:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1480.278768] LustreError: 78135:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1480.333284] Lustre: server umount lustre-MDT0000 complete [ 1482.662828] LustreError: 73310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810712 with bad export cookie 16303905360691149838 [ 1482.665765] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1482.675032] LustreError: 73310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1482.889729] Lustre: server umount lustre-MDT0001 complete [ 1487.320502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1493.431873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.639294] Lustre: 66506:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810714/real 1756810714] req@00000000415ef045 x1842149352097024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756810730 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1500.651878] Lustre: 66506:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1500.656377] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.662888] Lustre: Skipped 1 previous similar message [ 1507.873215] LustreError: 66504:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000050f325e4 x1842149352098048/t0(0) o250->MGC192.168.202.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1508.037844] LustreError: 74936:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.081183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1508.085467] Lustre: Skipped 1 previous similar message [ 1508.125318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1509.087249] Lustre: 66507:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810722/real 1756810722] req@000000008e66baca x1842149352097536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756810738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1509.105985] Lustre: 66507:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1510.181566] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1515.315235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1518.111537] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1520.622579] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1521.542271] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1523.678604] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1530.275074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1530.276380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1530.282889] Lustre: Skipped 1 previous similar message [ 1530.287376] Lustre: Skipped 2 previous similar messages [ 1533.334068] LustreError: 80593:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1533.336893] LustreError: 80593:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1533.408489] Lustre: server umount lustre-MDT0000 complete [ 1535.457323] LustreError: 79191:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.465397] LustreError: 79191:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 1535.480904] LustreError: 73310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810765 with bad export cookie 16303905360691151210 [ 1535.484386] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1535.486588] LustreError: 73310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1535.669613] Lustre: server umount lustre-MDT0001 complete [ 1538.633653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1540.184148] Lustre: DEBUG MARKER: files compared the same [ 1551.380322] Lustre: server umount lustre-OST0000 complete [ 1559.513633] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1560.928854] Key type lgssc unregistered [ 1561.095898] LNet: 82225:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.099985] LNetError: 82225:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.114535] LNet: Removed LNI 192.168.202.121@tcp [ 1561.538658] Key type .llcrypt unregistered [ 1561.539982] Key type ._llcrypt unregistered [ 1573.623283] Key type ._llcrypt registered [ 1573.625195] Key type .llcrypt registered [ 1573.697467] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1574.241198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1574.250033] alg: No test for adler32 (adler32-zlib) [ 1575.194771] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1575.351056] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1576.999528] Key type lgssc registered [ 1577.606194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1582.121689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1582.138642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.472515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.811117] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug +malloc all [ 1588.705680] LustreError: 83600:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1588.719604] LustreError: 83600:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1591.538856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1591.782102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1593.980312] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug +malloc all [ 1596.924256] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1598.576966] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1601.824864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1601.952547] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1601.987941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1604.702111] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug +malloc all [ 1607.069862] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1608.680618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1616.865039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1616.868645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.874871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1618.913548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.914039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1621.683402] LustreError: 85845:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1621.769885] Lustre: server umount lustre-OST0000 complete [ 1624.032144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.032770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1624.037927] Lustre: Skipped 1 previous similar message [ 1624.041109] Lustre: Skipped 1 previous similar message [ 1629.153187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.157786] Lustre: Skipped 2 previous similar messages [ 1638.367680] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1638.410910] LustreError: 86046:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1638.413875] LustreError: 86046:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1638.469569] Lustre: server umount lustre-MDT0000 complete [ 1639.394882] LustreError: 83600:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.402498] LustreError: 83600:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1640.130968] LustreError: 83579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810869 with bad export cookie 18084073376118469018 [ 1640.134650] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1640.136309] LustreError: 83579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1640.205359] LustreError: 86262:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1640.208563] LustreError: 86262:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1640.327655] Lustre: server umount lustre-MDT0001 complete [ 1645.500215] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1647.116665] Key type lgssc unregistered [ 1647.315805] LNet: 86794:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.322485] LNetError: 86794:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.338257] LNet: Removed LNI 192.168.202.121@tcp [ 1647.848486] Key type .llcrypt unregistered [ 1647.850267] Key type ._llcrypt unregistered [ 1660.797754] Key type ._llcrypt registered [ 1660.798929] Key type .llcrypt registered [ 1660.846163] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1661.223580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1661.249033] alg: No test for adler32 (adler32-zlib) [ 1662.146083] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1662.257424] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1663.855220] Key type lgssc registered [ 1664.383642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1668.213449] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1668.233908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1685.791388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1691.937835] LustreError: 88143:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 1721.631290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.254920] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1726.750972] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1730.294527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.365213] Lustre: *** cfs_fail_loc=706, val=0*** [ 1730.561271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.607509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1732.352355] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1735.649421] LustreError: 88169:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1735.656302] LustreError: 88169:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1736.060513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1736.267316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1737.938703] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1740.085445] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1741.594307] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1742.816983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.818903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1742.822543] Lustre: Skipped 1 previous similar message [ 1742.827208] Lustre: Skipped 1 previous similar message [ 1747.942859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.945605] Lustre: Skipped 1 previous similar message [ 1748.524185] LustreError: 90465:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1748.599571] Lustre: server umount lustre-OST0000 complete [ 1751.521042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1751.524633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.533051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.058450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.351095] LustreError: 90667:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1756.353126] LustreError: 90667:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1756.400821] Lustre: server umount lustre-MDT0000 complete [ 1758.072864] LustreError: 88150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810987 with bad export cookie 15948310659036797408 [ 1758.076065] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1758.080438] LustreError: 88150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1758.154656] LustreError: 90870:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1758.157611] LustreError: 90870:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1758.274081] Lustre: server umount lustre-MDT0001 complete [ 1762.738764] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1763.997591] Key type lgssc unregistered [ 1764.141547] LNet: 91401:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.145427] LNetError: 91401:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.152321] LNet: Removed LNI 192.168.202.121@tcp [ 1764.535655] Key type .llcrypt unregistered [ 1764.537285] Key type ._llcrypt unregistered [ 1775.676058] Key type ._llcrypt registered [ 1775.677625] Key type .llcrypt registered [ 1775.721710] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1776.167522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1776.200824] alg: No test for adler32 (adler32-zlib) [ 1777.094318] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1777.197463] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1778.799559] Key type lgssc registered [ 1779.287305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1782.582617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1782.592277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.275341] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1788.545957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1788.794178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1790.500117] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1794.016068] LustreError: 93446:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.022862] LustreError: 93446:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1794.199556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1794.364365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.947726] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1798.432896] LustreError: 93450:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1798.441895] LustreError: 93450:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1800.769956] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1800.778103] Lustre: MGS: Client 4b11ef2d-a5c4-4874-9aeb-4c18001810a8 (at 0@lo) reconnecting [ 1800.906768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.947090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1802.590921] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1811.427128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.428596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1811.432727] Lustre: Skipped 1 previous similar message [ 1811.433175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1811.436325] Lustre: Skipped 1 previous similar message [ 1816.545111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1816.547797] Lustre: Skipped 1 previous similar message [ 1817.070121] LustreError: 95407:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1817.144460] Lustre: server umount lustre-OST0000 complete [ 1821.665569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.666061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1821.675766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.679809] Lustre: Skipped 1 previous similar message [ 1824.710098] LustreError: 95610:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1824.713308] LustreError: 95610:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1824.725989] Lustre: server umount lustre-MDT0000 complete [ 1826.377831] LustreError: 95812:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1826.380246] LustreError: 95812:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1826.469953] Lustre: server umount lustre-MDT0001 complete [ 1828.091109] Lustre: server umount MGS complete [ 1832.640453] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:04:21 (1756811061) [ 1844.555217] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 1848.195631] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1851.990862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1854.517327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1856.710453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1859.139646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1861.794931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1861.825531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.381355] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1867.595548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1867.619237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1867.703673] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1867.722685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1867.763275] Lustre: lustre-MDT0001: new disk, initializing [ 1867.795106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1867.804588] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1867.807212] Lustre: 100523:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1867.810673] LustreError: 100523:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1869.276589] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1873.803802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1873.830704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1873.932446] Lustre: lustre-OST0000: new disk, initializing [ 1873.935361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1873.966411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.055617] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1878.368141] Lustre: 101714:0:(client.c:1628:after_reply()) @@@ resending request on EINPROGRESS req@00000000af8d7372 x1842149857461504/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1756811123 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1879.326160] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1881.971636] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1881.976703] Lustre: MGS: Client 52f031bc-e01f-4c89-8730-9c672d89449d (at 0@lo) reconnecting [ 1881.993856] Lustre: 99808:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1882.003672] LustreError: 99808:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1882.018942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1882.062956] Lustre: lustre-MDT0000: new disk, initializing [ 1882.138195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.151161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1883.168200] Lustre: 102234:0:(client.c:1628:after_reply()) @@@ resending request on EINPROGRESS req@000000005cc42472 x1842149857466752/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1756811128 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1883.231768] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1883.772817] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1887.459559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1887.463855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1888.226588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1888.293732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1908.191905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1908.195528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.200945] Lustre: Skipped 1 previous similar message [ 1908.203268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1912.289709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1912.292712] Lustre: Skipped 2 previous similar messages [ 1922.015162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1922.048106] LustreError: 102756:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1922.051530] LustreError: 102756:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1922.155194] Lustre: server umount lustre-OST0000 complete [ 1927.649405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.649420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1927.650142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.650147] Lustre: Skipped 2 previous similar messages [ 1927.654627] Lustre: Skipped 2 previous similar messages [ 1927.657776] LustreError: Skipped 1 previous similar message [ 1930.048355] LustreError: 102960:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1930.050398] LustreError: 102960:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1930.058843] Lustre: server umount lustre-MDT0000 complete [ 1937.686137] Lustre: server umount lustre-MDT0001 complete [ 1939.170980] LustreError: 103364:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1939.174354] LustreError: 103364:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1939.206393] Lustre: server umount MGS complete [ 1943.979383] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:06:13 (1756811173) [ 1951.154486] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1952.523252] Key type lgssc unregistered [ 1952.678681] LNet: 104700:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.682160] LNetError: 104700:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.689855] LNet: Removed LNI 192.168.202.121@tcp [ 1953.085935] Key type .llcrypt unregistered [ 1953.087685] Key type ._llcrypt unregistered [ 1961.501133] Key type ._llcrypt registered [ 1961.502787] Key type .llcrypt registered [ 1961.553409] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1961.951221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1961.985120] alg: No test for adler32 (adler32-zlib) [ 1962.867674] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1962.974689] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1964.567190] Key type lgssc registered [ 1965.055756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1969.011946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1969.028217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.295282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.141974] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1975.880062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1976.084223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1977.938916] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1980.236933] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1981.753563] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1984.895586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1985.073534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.672747] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1990.029840] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1995.313708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2000.352672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2000.353794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.355780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2000.364084] Lustre: Skipped 1 previous similar message [ 2004.660342] LustreError: 108125:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2004.753476] Lustre: server umount lustre-OST0000 complete [ 2007.008683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2007.011943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.017620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.022851] Lustre: Skipped 1 previous similar message [ 2010.592584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.593222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.766456] LustreError: 108327:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2012.769283] LustreError: 108327:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2012.828593] Lustre: server umount lustre-MDT0000 complete [ 2014.454581] LustreError: 107497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811244 with bad export cookie 4051685726840607128 [ 2014.457936] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2014.459495] LustreError: 107497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2014.530139] LustreError: 108528:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2014.533210] LustreError: 108528:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2014.623735] Lustre: server umount lustre-MDT0001 complete [ 2018.807583] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2020.133797] Key type lgssc unregistered [ 2020.299867] LNet: 109059:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.303698] LNetError: 109059:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2020.314332] LNet: Removed LNI 192.168.202.121@tcp [ 2020.723975] Key type .llcrypt unregistered [ 2020.725673] Key type ._llcrypt unregistered [ 2027.709768] Key type ._llcrypt registered [ 2027.711045] Key type .llcrypt registered [ 2027.754446] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2028.211684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2028.242952] alg: No test for adler32 (adler32-zlib) [ 2029.122126] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2029.219223] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2030.815173] Key type lgssc registered [ 2031.348218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2034.241177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2034.241423] LustreError: 110100:0:(tgt_mount.c:2416:server_fill_super()) cfs_race id 716 sleeping [ 2034.244397] Lustre: Skipped 1 previous similar message [ 2034.244420] LustreError: 110099:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 waking [ 2034.252964] LustreError: 110100:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 2034.256395] LustreError: 110100:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2034.263283] LustreError: 110100:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2034.271068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.525828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.053136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2039.203365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2040.910948] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2043.233542] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing lsmod [ 2044.592481] LustreError: 111276:0:(tgt_mount.c:2416:server_fill_super()) cfs_race id 716 sleeping [ 2044.592653] LustreError: 111275:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 waking [ 2044.598781] LustreError: 111276:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2044.601464] LustreError: 111276:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2044.606843] LustreError: 111276:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2044.611186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2044.807864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.874841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2049.252882] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.255705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.261933] Lustre: Skipped 1 previous similar message [ 2052.291060] LustreError: 111466:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2052.332258] Lustre: server umount lustre-MDT0000 complete [ 2053.934421] LustreError: 110119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811283 with bad export cookie 15202924618770878220 [ 2053.937129] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2053.939539] LustreError: 110119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2054.624271] LustreError: 110135:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.624626] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2054.630752] LustreError: 110135:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2054.636129] Lustre: Skipped 1 previous similar message [ 2054.642395] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2054.645024] Lustre: Skipped 2 previous similar messages [ 2055.391326] LustreError: 110572:0:(client.c:1371:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008d6092a0 x1842150121686528/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2055.404928] LustreError: 111669:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2055.407716] LustreError: 111669:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2055.496405] Lustre: server umount lustre-MDT0001 complete [ 2057.186890] LustreError: 111871:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2057.189547] LustreError: 111871:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2057.273914] Lustre: server umount lustre-OST0000 complete [ 2061.102487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.328723] LustreError: 112347:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.364809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.007982] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2066.400623] LustreError: 112347:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.505783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2066.667673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2068.394060] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2070.729638] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2072.277808] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2075.303227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2075.473137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.068502] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2080.370407] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2085.738386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 2096.098484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.099826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2096.103444] Lustre: Skipped 1 previous similar message [ 2096.108413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2098.606463] LustreError: 114616:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2098.609024] LustreError: 114616:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2098.679689] Lustre: server umount lustre-OST0000 complete [ 2101.216943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.217546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.222199] Lustre: Skipped 1 previous similar message [ 2101.224733] Lustre: Skipped 1 previous similar message [ 2106.304340] LustreError: 114817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2106.306920] LustreError: 114817:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2106.335922] LustreError: 112346:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.336892] Lustre: server umount lustre-MDT0000 complete [ 2106.342865] LustreError: 112346:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2107.763589] LustreError: 112326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811337 with bad export cookie 15202924618770879137 [ 2107.766938] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2107.769115] LustreError: 112326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2107.925606] Lustre: server umount lustre-MDT0001 complete [ 2112.121456] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2113.357688] Key type lgssc unregistered [ 2113.505490] LNet: 115550:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.508517] LNetError: 115550:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.517494] LNet: Removed LNI 192.168.202.121@tcp [ 2113.881479] Key type .llcrypt unregistered [ 2113.883144] Key type ._llcrypt unregistered [ 2124.080319] Key type ._llcrypt registered [ 2124.081755] Key type .llcrypt registered [ 2124.124744] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2124.626907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2124.637856] alg: No test for adler32 (adler32-zlib) [ 2125.509020] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2125.615068] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2127.215261] Key type lgssc registered [ 2127.664588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2130.835590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2130.844626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.050577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.550683] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2136.848639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2137.026738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2138.486135] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2140.377750] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2141.852449] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2144.518069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2144.657724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.741894] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2148.797439] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2156.969450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2161.646619] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2162.808698] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2167.267089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2167.267271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.270267] LustreError: Skipped 1 previous similar message [ 2167.271173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2167.275976] Lustre: Skipped 1 previous similar message [ 2170.092132] LustreError: 119295:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2170.159428] Lustre: server umount lustre-OST0000 complete [ 2172.385125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.385706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2172.389707] Lustre: Skipped 1 previous similar message [ 2172.392238] Lustre: Skipped 1 previous similar message [ 2177.504697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.507455] Lustre: Skipped 1 previous similar message [ 2177.732293] LustreError: 119500:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2177.734498] LustreError: 119500:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2177.775137] Lustre: server umount lustre-MDT0000 complete [ 2179.374440] LustreError: 119149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811409 with bad export cookie 17852001644396592754 [ 2179.376752] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2179.379073] LustreError: 119149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2179.457025] LustreError: 119702:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2179.459817] LustreError: 119702:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2179.557094] Lustre: server umount lustre-MDT0001 complete [ 2183.755310] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2184.985570] Key type lgssc unregistered [ 2185.146419] LNet: 120233:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.149683] LNetError: 120233:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.161231] LNet: Removed LNI 192.168.202.121@tcp [ 2185.536474] Key type .llcrypt unregistered [ 2185.538233] Key type ._llcrypt unregistered [ 2193.896304] Key type ._llcrypt registered [ 2193.898024] Key type .llcrypt registered [ 2193.959507] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2194.465940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2194.504351] alg: No test for adler32 (adler32-zlib) [ 2195.384744] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2195.493609] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2197.103179] Key type lgssc registered [ 2197.573832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2200.857376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2200.877333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.113100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.776563] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2207.302885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2207.487099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2209.170718] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2211.345606] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2212.778637] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2215.742276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2215.922230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.326985] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2220.562708] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2227.174127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2237.408837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2237.410739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2237.418110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2237.420717] Lustre: Skipped 1 previous similar message [ 2240.301700] LustreError: 123681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2240.368570] Lustre: server umount lustre-OST0000 complete [ 2242.528896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.529483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.533931] Lustre: Skipped 2 previous similar messages [ 2242.537817] Lustre: Skipped 1 previous similar message [ 2247.648183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.650919] Lustre: Skipped 1 previous similar message [ 2252.768271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.770602] Lustre: Skipped 1 previous similar message [ 2256.351224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2256.393712] LustreError: 123882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2256.396880] LustreError: 123882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2256.428748] Lustre: server umount lustre-MDT0000 complete [ 2257.888363] LustreError: 121414:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.895561] LustreError: 121414:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2257.965209] LustreError: 121395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811487 with bad export cookie 17087856311825745834 [ 2257.969500] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.970918] LustreError: 121395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2258.047593] LustreError: 124085:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2258.051087] LustreError: 124085:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2258.141642] Lustre: server umount lustre-MDT0001 complete [ 2262.349589] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2263.555198] Key type lgssc unregistered [ 2263.708205] LNet: 124616:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.711342] LNetError: 124616:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.721374] LNet: Removed LNI 192.168.202.121@tcp [ 2264.073362] Key type .llcrypt unregistered [ 2264.075095] Key type ._llcrypt unregistered [ 2273.712491] Key type ._llcrypt registered [ 2273.714172] Key type .llcrypt registered [ 2273.758409] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2274.188670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.205113] alg: No test for adler32 (adler32-zlib) [ 2275.073983] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2275.170137] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2276.759210] Key type lgssc registered [ 2277.171170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2280.059350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2280.071110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.252541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.702992] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2285.782703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2285.909367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2287.333762] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2289.239343] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2290.521206] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2293.156765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2293.309158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.530262] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2297.565153] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2305.644917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 2310.667260] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2312.070134] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2312.073457] Lustre: Skipped 2 previous similar messages [ 2313.797630] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2313.800610] Lustre: Skipped 2 previous similar messages [ 2319.881498] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2319.894598] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2323.727352] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.21@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2323.732024] Lustre: Skipped 2 previous similar messages [ 2328.584284] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.21@tcp 192.168.0.[2,10]@tcp [ 2334.464370] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.21/24@tcp in log lustre-MDT0000 [ 2334.468588] Lustre: Skipped 2 previous similar messages [ 2334.730481] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.21/24@tcp [ 2334.733220] Lustre: Skipped 1 previous similar message [ 2346.979166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.979315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2346.980039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2346.984435] Lustre: Skipped 1 previous similar message [ 2346.991609] LustreError: Skipped 1 previous similar message [ 2348.652165] LustreError: 129712:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2348.725684] Lustre: server umount lustre-OST0000 complete [ 2352.096702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.097546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2352.102734] Lustre: Skipped 1 previous similar message [ 2352.105163] Lustre: Skipped 1 previous similar message [ 2356.299696] LustreError: 129913:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2356.302709] LustreError: 129913:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2356.331523] Lustre: server umount lustre-MDT0000 complete [ 2357.217473] LustreError: 128438:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.224492] LustreError: 128438:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2357.657089] LustreError: 125973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811587 with bad export cookie 1003041098846303005 [ 2357.659593] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2357.661231] LustreError: 125973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2357.722973] LustreError: 130114:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2357.725314] LustreError: 130114:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2357.814060] Lustre: server umount lustre-MDT0001 complete [ 2361.667816] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2362.785539] Key type lgssc unregistered [ 2362.930381] LNet: 130645:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.933124] LNetError: 130645:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2362.941322] LNet: Removed LNI 192.168.202.121@tcp [ 2363.237518] Key type .llcrypt unregistered [ 2363.238644] Key type ._llcrypt unregistered [ 2373.798045] Key type ._llcrypt registered [ 2373.799411] Key type .llcrypt registered [ 2373.838622] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2374.147681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.160272] alg: No test for adler32 (adler32-zlib) [ 2375.024735] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2375.123053] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2376.711152] Key type lgssc registered [ 2377.063832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2379.703097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2379.710366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.852286] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2380.858095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.861587] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.21/24@tcp [ 2382.176445] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2385.245205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2385.357162] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2385.365758] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.21/24@tcp [ 2385.382695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2386.805254] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2388.725053] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2389.951812] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2392.306173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2392.423169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.363527] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2396.434879] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2405.735888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 2409.952113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2409.955173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.959605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.977128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.979265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.979736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.980159] LustreError: 134486:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2416.060112] Lustre: server umount lustre-OST0000 complete [ 2421.217605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.218238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2421.218589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.218595] Lustre: Skipped 1 previous similar message [ 2421.223367] Lustre: Skipped 1 previous similar message [ 2423.744545] LustreError: 134688:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2423.747722] LustreError: 134688:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2423.782668] Lustre: server umount lustre-MDT0000 complete [ 2425.172315] LustreError: 132143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811654 with bad export cookie 1826968016520703874 [ 2425.174729] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.176892] LustreError: 132143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2425.243814] LustreError: 134891:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2425.246822] LustreError: 134891:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2425.327290] Lustre: server umount lustre-MDT0001 complete [ 2429.195681] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2430.438315] Key type lgssc unregistered [ 2430.597832] LNet: 135422:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2430.602548] LNetError: 135422:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2430.609521] LNet: Removed LNI 192.168.202.121@tcp [ 2430.958534] Key type .llcrypt unregistered [ 2430.959926] Key type ._llcrypt unregistered [ 2440.324641] Key type ._llcrypt registered [ 2440.325857] Key type .llcrypt registered [ 2440.365052] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2446.795106] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2447.172067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.222052] alg: No test for adler32 (adler32-zlib) [ 2448.110814] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2448.198577] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2449.783294] Key type lgssc registered [ 2450.356748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2453.925827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2456.379171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2458.651511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2461.140589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2463.973489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2463.996353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2464.003785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.103992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2465.126237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2465.169232] Lustre: lustre-MDT0000: new disk, initializing [ 2465.219527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.230257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2466.740317] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2471.299864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2471.322864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2471.363944] Lustre: 138770:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2471.391088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2471.393775] Lustre: Skipped 1 previous similar message [ 2471.433220] Lustre: lustre-MDT0001: new disk, initializing [ 2471.472389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2471.487870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2471.491680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2472.994229] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2476.519311] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2477.850476] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2480.326597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2480.350770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2480.455626] Lustre: lustre-OST0000: new disk, initializing [ 2480.458396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2480.487316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.470209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2481.474279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2481.506709] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2482.481886] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2485.855420] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2487.309444] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2487.397352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2488.847505] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2488.933920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2491.693210] LustreError: 141708:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2491.753664] Lustre: server umount lustre-OST0000 complete [ 2496.994847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.995235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2496.995542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2496.999697] Lustre: Skipped 1 previous similar message [ 2502.113443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2502.115701] Lustre: Skipped 2 previous similar messages [ 2503.139879] LustreError: 141908:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2503.142106] LustreError: 141908:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2503.173333] Lustre: server umount lustre-MDT0000 complete [ 2504.510121] LustreError: 138763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811734 with bad export cookie 17996838486597438229 [ 2504.512821] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.514126] LustreError: 138763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2514.923980] LustreError: 142112:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2514.926735] LustreError: 142112:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2515.020607] Lustre: server umount lustre-MDT0001 complete [ 2523.213521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.441499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.909380] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2528.273502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2528.424760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2529.974951] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2532.124978] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2533.609711] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2536.370637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2536.522682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.869241] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2541.189240] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2542.832551] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2549.086666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 2550.555627] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2550.650678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2554.491939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2554.515166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2554.574543] Lustre: lustre-OST0001: new disk, initializing [ 2554.576864] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2554.607161] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2556.704819] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2560.119900] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2560.514717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2560.518249] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2560.550543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2561.227623] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2562.548541] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2562.625338] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2563.868492] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff942ad8f71800.ost_server_uuid 50 [ 2564.399311] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff942ad8f71800.ost_server_uuid in FULL state after 0 sec [ 2570.723216] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.724514] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.726456] LustreError: Skipped 1 previous similar message [ 2570.730553] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2570.733893] Lustre: Skipped 1 previous similar message [ 2572.589514] LustreError: 146886:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2572.592826] LustreError: 146886:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2572.610657] Lustre: server umount lustre-OST0001 complete [ 2574.816934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2574.816974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.819233] LustreError: Skipped 1 previous similar message [ 2574.824842] Lustre: Skipped 1 previous similar message [ 2574.827357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2574.830022] Lustre: Skipped 1 previous similar message [ 2575.841182] LustreError: 144186:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.846034] LustreError: 144186:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2580.013991] LustreError: 147087:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2580.017252] LustreError: 147087:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2580.081782] Lustre: server umount lustre-OST0000 complete [ 2585.056035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2585.058743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.062736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2585.065356] Lustre: Skipped 2 previous similar messages [ 2587.549457] Lustre: server umount lustre-MDT0000 complete [ 2588.806242] LustreError: 144207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811818 with bad export cookie 17996838486597439104 [ 2588.809948] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2588.811909] LustreError: 144207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2588.875297] LustreError: 147490:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2588.877052] LustreError: 147490:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2588.985780] Lustre: server umount lustre-MDT0001 complete [ 2595.728821] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2596.859188] Key type lgssc unregistered [ 2596.992402] LNet: 148478:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2596.994844] LNetError: 148478:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2597.002271] LNet: Removed LNI 192.168.202.121@tcp [ 2597.307963] Key type .llcrypt unregistered [ 2597.309562] Key type ._llcrypt unregistered [ 2608.985222] Key type ._llcrypt registered [ 2608.986741] Key type .llcrypt registered [ 2609.031269] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2615.130570] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2615.503990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2615.529734] alg: No test for adler32 (adler32-zlib) [ 2616.390559] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2616.479046] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2618.063188] Key type lgssc registered [ 2618.428559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2621.767502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2624.113978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2626.287751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2628.584446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2630.956710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2630.979247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2630.984736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.074664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2632.089688] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2632.126284] Lustre: lustre-MDT0000: new disk, initializing [ 2632.154569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.161470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2633.631334] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2638.008209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2638.036141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2638.071596] Lustre: 152412:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2638.101236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2638.103585] Lustre: Skipped 1 previous similar message [ 2638.140585] Lustre: lustre-MDT0001: new disk, initializing [ 2638.180700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2638.195059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2638.198905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2639.648362] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2642.948559] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2644.209857] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2646.718567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2646.752337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2646.866470] Lustre: lustre-OST0000: new disk, initializing [ 2646.869363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2646.903560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.077899] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2652.595665] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2654.217348] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2654.719689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2654.726191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2654.775319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2655.352244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2657.044958] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2657.165306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2659.811046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.812531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2659.816711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.818767] LustreError: Skipped 1 previous similar message [ 2659.820983] Lustre: Skipped 1 previous similar message [ 2664.112408] LustreError: 155359:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2664.175882] Lustre: server umount lustre-OST0000 complete [ 2669.024284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2669.027122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.031821] Lustre: Skipped 1 previous similar message [ 2669.034383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.049101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.050426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.682697] LustreError: 155561:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2671.684882] LustreError: 155561:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2671.728790] Lustre: server umount lustre-MDT0000 complete [ 2673.138346] LustreError: 153195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811902 with bad export cookie 1949391497899245050 [ 2673.141679] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2673.142841] LustreError: 153195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2673.213715] LustreError: 155763:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2673.216298] LustreError: 155763:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2673.318058] Lustre: server umount lustre-MDT0001 complete [ 2676.294203] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 07:18:25 (1756811905) [ 2681.875989] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2685.183117] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2688.584065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2690.895867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2693.210832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2695.502271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2697.932685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2697.959149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.043257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2698.055771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2698.096670] Lustre: lustre-MDT0000: new disk, initializing [ 2698.126469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.135685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2699.505519] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2703.913204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2703.944102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2703.979327] Lustre: 158748:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2704.005558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2704.008671] Lustre: Skipped 1 previous similar message [ 2704.044393] Lustre: lustre-MDT0001: new disk, initializing [ 2704.085161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2704.100333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2704.104574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2705.494379] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2708.568638] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2709.749710] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2711.992612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2712.012816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2712.115383] Lustre: lustre-OST0000: new disk, initializing [ 2712.117963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2712.144801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.085676] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2716.661680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2716.665723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2716.695759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2717.254199] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2719.231742] Lustre: Failing over lustre-OST0000 [ 2719.258697] LustreError: 161196:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2719.261780] LustreError: 161196:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2719.334102] Lustre: server umount lustre-OST0000 complete [ 2732.185737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2732.289596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.296740] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2733.892865] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2734.244840] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2735.497154] Lustre: Failing over lustre-MDT0000 [ 2735.557783] LustreError: 162171:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2735.559466] LustreError: 162171:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2735.588403] Lustre: server umount lustre-MDT0000 complete [ 2737.567144] Lustre: 150632:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756811951/real 1756811951] req@000000004c67d175 x1842150738287872/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1756811967 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2737.576997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.687152] Lustre: 150631:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756811956/real 1756811956] req@00000000c1b2de5f x1842150738289920/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1756811972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2742.752342] LustreError: 159535:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.757930] LustreError: 159535:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 7 previous similar messages [ 2744.122622] LustreError: 158783:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.871106] Lustre: 150632:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756811961/real 1756811961] req@00000000369009be x1842150738291072/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1756811977 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2747.873888] LustreError: 158753:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.887843] LustreError: 158753:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2748.653207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.694889] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2748.699698] Lustre: 158750:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.121@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2748.777020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2748.830484] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2749.242922] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2750.226358] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2754.025822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2754.033042] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2754.036672] Lustre: Skipped 1 previous similar message [ 2754.052722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2759.136965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2759.138556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.140869] LustreError: Skipped 1 previous similar message [ 2759.144132] Lustre: Skipped 3 previous similar messages [ 2759.144839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.149037] Lustre: Skipped 2 previous similar messages [ 2761.262094] LustreError: 163054:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2761.264733] LustreError: 163054:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2761.343539] Lustre: server umount lustre-OST0000 complete [ 2764.256715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2764.257749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2764.261009] Lustre: Skipped 1 previous similar message [ 2768.788427] Lustre: server umount lustre-MDT0000 complete [ 2769.376038] LustreError: 158759:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.382619] LustreError: 158759:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2770.114397] LustreError: 158739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811999 with bad export cookie 1949391497899247122 [ 2770.116861] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.118670] LustreError: 158739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2770.263349] Lustre: server umount lustre-MDT0001 complete [ 2774.098804] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2775.295443] Key type lgssc unregistered [ 2775.439501] LNet: 163987:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2775.442373] LNetError: 163987:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2775.454619] LNet: Removed LNI 192.168.202.121@tcp [ 2775.794323] Key type .llcrypt unregistered [ 2775.795640] Key type ._llcrypt unregistered [ 2796.231319] Key type ._llcrypt registered [ 2796.232859] Key type .llcrypt registered [ 2796.280247] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2796.733863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2796.817353] alg: No test for adler32 (adler32-zlib) [ 2797.678575] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2797.774452] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2799.367427] Key type lgssc registered [ 2799.796623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2802.631584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2802.639536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.805238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.291896] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2808.524929] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2810.475063] LustreError: 165517:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2810.479940] LustreError: 165517:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2810.483899] LustreError: 165517:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2810.542566] LustreError: 166039:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2810.625312] Lustre: server umount lustre-MDT0000 complete [ 2815.513530] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2818.825382] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2822.488987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2824.820286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2827.087130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2829.489385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2832.016209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2832.040512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.157432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2832.172625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2832.212350] Lustre: lustre-MDT0000: new disk, initializing [ 2832.240496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.248718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2833.637038] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2838.038989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.059357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2838.097196] Lustre: 168677:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2838.121132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2838.123407] Lustre: Skipped 1 previous similar message [ 2838.162222] Lustre: lustre-MDT0001: new disk, initializing [ 2838.202849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2838.217722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2838.221763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2839.618677] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2842.873383] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2844.153778] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2846.426586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2846.452227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2846.554810] Lustre: lustre-OST0000: new disk, initializing [ 2846.557345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2846.588479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.519839] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2851.727692] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2854.902080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2854.905637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2854.933724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2859.036826] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2859.616274] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:21:28 (1756812088) [ 2936.799601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.799782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2936.804631] Lustre: Skipped 2 previous similar messages [ 2936.807808] Lustre: Skipped 1 previous similar message [ 2941.919647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2941.922036] Lustre: Skipped 2 previous similar messages [ 2942.587682] LustreError: 172181:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2942.589729] LustreError: 172181:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2942.640825] Lustre: server umount lustre-MDT0000 complete [ 2943.951555] LustreError: 170422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812173 with bad export cookie 9116941521063443562 [ 2943.952573] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2943.956045] LustreError: 170422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2947.039683] LustreError: 168683:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.039701] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.045342] LustreError: 168683:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2947.049694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2948.063427] LustreError: 168685:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.183472] LustreError: 172078:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.183478] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2953.191664] Lustre: Skipped 1 previous similar message [ 2958.303102] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2958.336614] LustreError: 172383:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2958.338479] LustreError: 172383:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2958.403095] Lustre: server umount lustre-MDT0001 complete [ 2974.175154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2974.193625] LustreError: 172585:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2974.196840] LustreError: 172585:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2974.260903] Lustre: server umount lustre-OST0000 complete [ 2977.666060] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2981.039410] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2984.906107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2987.477427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2989.872418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.243598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2994.900371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2994.938372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.036563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2995.053334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2995.094573] Lustre: lustre-MDT0000: new disk, initializing [ 2995.133265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.143190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2996.687032] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3001.156692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3001.181429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3001.219347] Lustre: 174922:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3001.251203] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3001.251530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3001.253845] Lustre: Skipped 1 previous similar message [ 3001.258065] Lustre: Skipped 1 previous similar message [ 3001.296511] Lustre: lustre-MDT0001: new disk, initializing [ 3001.351860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3001.356307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3002.949375] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3006.219446] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3007.530768] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3009.975260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3010.008699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3010.112792] Lustre: lustre-OST0000: new disk, initializing [ 3010.114719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3010.143194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.145049] Lustre: Skipped 1 previous similar message [ 3010.235357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3010.240246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3010.277934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3012.135679] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3015.431469] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3016.839299] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3016.934308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3018.302992] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3018.386738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3020.768948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3020.769804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.772380] LustreError: Skipped 1 previous similar message [ 3020.778170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3020.782060] Lustre: Skipped 1 previous similar message [ 3021.103128] LustreError: 177860:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3021.105118] LustreError: 177860:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3021.175490] Lustre: server umount lustre-OST0000 complete [ 3025.889023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.893816] Lustre: Skipped 2 previous similar messages [ 3028.701419] Lustre: server umount lustre-MDT0000 complete [ 3030.026831] LustreError: 174915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812259 with bad export cookie 9116941521063602378 [ 3030.031296] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3030.031503] LustreError: 174915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3030.096398] LustreError: 178263:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3030.098827] LustreError: 178263:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3030.193589] Lustre: server umount lustre-MDT0001 complete [ 3033.138407] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:24:22 (1756812262) [ 3038.760703] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3042.013904] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3045.427090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3047.713240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3049.799743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3052.023261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3054.325699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3054.349443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.428552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3054.439691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3054.473429] Lustre: lustre-MDT0000: new disk, initializing [ 3054.499632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.506554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3055.859195] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3060.167803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3060.190505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3060.232061] Lustre: 181248:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3060.331595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3061.715356] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3064.764036] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3065.950833] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3068.270593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3068.297563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3070.471956] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3070.966954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3071.005588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3073.738961] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3076.066023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3076.066978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.069489] LustreError: Skipped 1 previous similar message [ 3076.070684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3076.073921] Lustre: Skipped 1 previous similar message [ 3076.075524] Lustre: Skipped 2 previous similar messages [ 3081.776119] LustreError: 183792:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3081.778291] LustreError: 183792:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3081.838837] Lustre: server umount lustre-OST0000 complete [ 3085.792154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3085.794944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.299029] Lustre: server umount lustre-MDT0000 complete [ 3090.611192] LustreError: 181238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812320 with bad export cookie 9116941521063603232 [ 3090.613654] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3090.615939] LustreError: 181238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3090.764774] Lustre: server umount lustre-MDT0001 complete [ 3093.823334] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:25:22 (1756812322) [ 3099.237740] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3102.409092] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3105.961436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3108.271360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3110.455309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3112.771388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3115.190037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3115.211465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.290712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3115.303158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3115.305398] Lustre: Skipped 3 previous similar messages [ 3115.339231] Lustre: lustre-MDT0000: new disk, initializing [ 3115.340312] Lustre: Skipped 2 previous similar messages [ 3115.364782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.366573] Lustre: Skipped 2 previous similar messages [ 3115.371918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3115.374162] Lustre: Skipped 2 previous similar messages [ 3116.843164] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3121.211845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3121.233898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3121.270586] Lustre: 187180:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3121.371359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3122.687831] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3125.867585] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3127.117677] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3129.571051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3129.593827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3130.411535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3131.678200] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3134.720587] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3140.578053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3140.578983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.580666] LustreError: Skipped 1 previous similar message [ 3140.583579] Lustre: Skipped 1 previous similar message [ 3140.584264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.588078] Lustre: Skipped 7 previous similar messages [ 3142.702142] LustreError: 189728:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3142.704514] LustreError: 189728:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3142.784529] Lustre: server umount lustre-OST0000 complete [ 3150.163493] Lustre: server umount lustre-MDT0000 complete [ 3150.816337] LustreError: 187191:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.822337] LustreError: 187191:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3151.455078] LustreError: 187173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812381 with bad export cookie 9116941521063604303 [ 3151.458298] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3151.459019] LustreError: 187173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3151.601366] Lustre: server umount lustre-MDT0001 complete [ 3155.196208] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3156.245594] Key type lgssc unregistered [ 3156.381454] LNet: 190662:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3156.385128] LNetError: 190662:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3156.393587] LNet: Removed LNI 192.168.202.121@tcp [ 3156.689770] Key type .llcrypt unregistered [ 3156.691041] Key type ._llcrypt unregistered [ 3165.876381] Key type ._llcrypt registered [ 3165.877456] Key type .llcrypt registered [ 3165.910593] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3166.192154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3166.199431] alg: No test for adler32 (adler32-zlib) [ 3167.041665] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 3167.124228] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3168.703142] Key type lgssc registered [ 3169.046569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3171.605837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3171.613705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.766029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.159444] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3176.988381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3177.102426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3178.362542] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3180.183765] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3181.342459] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3183.621851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3183.732030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.603211] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3187.439927] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3199.078626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3214.816602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3214.818248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.820475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3214.824097] Lustre: Skipped 1 previous similar message [ 3219.939257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3219.941599] Lustre: Skipped 2 previous similar messages [ 3220.459893] LustreError: 194306:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3220.516895] Lustre: server umount lustre-OST0000 complete [ 3223.519888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3223.522791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.527845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3225.057081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.176800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.178947] Lustre: Skipped 3 previous similar messages [ 3235.296253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3235.298409] Lustre: Skipped 1 previous similar message [ 3236.319117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3236.350241] LustreError: 194508:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3236.352183] LustreError: 194508:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3236.381317] Lustre: server umount lustre-MDT0000 complete [ 3237.630453] LustreError: 193653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812467 with bad export cookie 7928988474658687025 [ 3237.633038] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3237.633787] LustreError: 193653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3237.702136] LustreError: 194711:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3237.704110] LustreError: 194711:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3237.780166] Lustre: server umount lustre-MDT0001 complete [ 3241.497642] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3242.627590] Key type lgssc unregistered [ 3242.773490] LNet: 195243:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.776943] LNetError: 195243:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.783522] LNet: Removed LNI 192.168.202.121@tcp [ 3243.109398] Key type .llcrypt unregistered [ 3243.111187] Key type ._llcrypt unregistered [ 3251.917893] Key type ._llcrypt registered [ 3251.919008] Key type .llcrypt registered [ 3251.957655] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3252.474282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3252.485204] alg: No test for adler32 (adler32-zlib) [ 3253.335541] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 3253.420227] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3254.999181] Key type lgssc registered [ 3255.379861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3257.884478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3257.891577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.036798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3260.292946] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3263.102506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3263.227064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3264.484824] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3266.339526] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3267.621603] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3270.043667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3270.189144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.272039] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3274.133238] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3283.490859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3288.547470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3288.549583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.550670] LustreError: Skipped 1 previous similar message [ 3288.556311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3288.559739] Lustre: Skipped 1 previous similar message [ 3289.924218] Lustre: lustre-OST0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 3293.664569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.666745] Lustre: Skipped 1 previous similar message [ 3298.785706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3298.788135] Lustre: Skipped 2 previous similar messages [ 3301.343157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3301.363649] LustreError: 198881:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3301.445784] Lustre: server umount lustre-OST0000 complete [ 3303.410810] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff942ac9160000.ost_server_uuid 50 [ 3303.975885] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff942ac9160000.ost_server_uuid in DISCONN state after 0 sec [ 3304.588834] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3314.655881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3314.658722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.664162] Lustre: Skipped 1 previous similar message [ 3314.666680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3314.669182] Lustre: Skipped 1 previous similar message [ 3319.776485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.697654] LustreError: 199280:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3320.699610] LustreError: 199280:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3320.724428] Lustre: server umount lustre-MDT0000 complete [ 3321.908817] LustreError: 196598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812551 with bad export cookie 8174161980085326717 [ 3321.913751] LustreError: 196598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3321.914553] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3321.977929] LustreError: 199484:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3321.980147] LustreError: 199484:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3322.083715] Lustre: server umount lustre-MDT0001 complete [ 3324.850279] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:29:14 (1756812554) [ 3326.996284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.140744] LustreError: 200153:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3327.147971] LustreError: 200153:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3327.164904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.441656] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3331.324604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3331.436554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3332.781810] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3334.576916] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3335.783409] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3338.024514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3339.888081] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3341.587505] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3343.772681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3343.799144] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3343.851113] Lustre: lustre-OST0001: new disk, initializing [ 3343.852611] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3343.879184] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3343.880932] Lustre: Skipped 1 previous similar message [ 3344.893198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3344.896639] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3344.924247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3345.655783] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3348.579908] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3349.987411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.990792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3349.992863] Lustre: Skipped 2 previous similar messages [ 3355.436254] LustreError: 203360:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3355.438418] LustreError: 203360:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3355.455847] Lustre: server umount lustre-OST0000 complete [ 3357.156709] LustreError: 203361:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.162229] LustreError: 203361:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3357.567117] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3357.654227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3359.036086] LustreError: 202852:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.079029] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3359.164790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3362.279586] LustreError: 202852:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.284734] LustreError: 202852:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3367.395130] LustreError: 202852:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.400364] LustreError: 202852:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3370.464146] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3370.464371] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3370.467213] LustreError: Skipped 1 previous similar message [ 3370.472626] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3370.475802] Lustre: Skipped 4 previous similar messages [ 3375.340140] LustreError: 204062:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3375.343680] LustreError: 204062:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3375.415418] Lustre: server umount lustre-OST0001 complete [ 3377.633837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3383.002843] Lustre: server umount lustre-MDT0000 complete [ 3384.353515] LustreError: 200133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812613 with bad export cookie 8174161980085327774 [ 3384.357070] LustreError: 200133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3384.357282] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3384.433249] LustreError: 204464:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3384.435533] LustreError: 204464:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3384.520461] Lustre: server umount lustre-MDT0001 complete [ 3386.514952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3387.897743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3389.279801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3390.682237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3393.018893] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:30:22 (1756812622) [ 3395.390807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.491961] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3395.505668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3395.590387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.052278] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3400.096912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3400.129808] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3400.145517] Lustre: 205954:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3401.667319] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3403.608244] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3404.883652] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3407.267354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3407.336167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3409.446554] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3411.342400] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3413.521960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3413.771276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3413.797202] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3415.920467] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3417.764668] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3419.109103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.110255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.114016] Lustre: Skipped 4 previous similar messages [ 3419.115321] Lustre: Skipped 6 previous similar messages [ 3419.660290] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3424.748085] LustreError: 209134:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3424.750704] LustreError: 209134:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3424.769656] Lustre: server umount lustre-OST0000 complete [ 3428.154497] LustreError: 207581:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3428.159487] LustreError: 207581:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3440.097598] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3440.099035] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.099837] LustreError: Skipped 1 previous similar message [ 3441.439493] Lustre: server umount lustre-OST0001 complete [ 3448.791097] Lustre: server umount lustre-MDT0000 complete [ 3449.992109] LustreError: 205992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812679 with bad export cookie 8174161980085328936 [ 3449.994667] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3449.994926] LustreError: 205992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3450.138183] Lustre: server umount lustre-MDT0001 complete [ 3451.978259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3453.181383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3454.372316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3455.635223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3457.763728] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:31:26 (1756812686) [ 3462.903495] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3465.857409] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3469.452619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3471.634047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.687094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3475.755266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3478.021058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3478.044622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.123442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3478.134875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3478.169699] Lustre: lustre-MDT0000: new disk, initializing [ 3478.193066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.196118] Lustre: Skipped 3 previous similar messages [ 3478.202404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3479.517529] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3483.583589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3483.615178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3483.645086] Lustre: 213566:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3483.668520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3483.670310] Lustre: Skipped 1 previous similar message [ 3483.709406] Lustre: lustre-MDT0001: new disk, initializing [ 3483.760859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3483.764183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3485.077799] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3488.106322] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3489.304724] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3491.539425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3491.563661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3491.650445] Lustre: lustre-OST0000: new disk, initializing [ 3491.653208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3493.522606] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3496.440593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3496.443849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3496.479731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3496.628760] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3497.901980] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3497.969729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3499.183206] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3499.257313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3501.538233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.538297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3501.541990] Lustre: Skipped 3 previous similar messages [ 3501.545156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3501.547391] Lustre: Skipped 9 previous similar messages [ 3505.963092] LustreError: 216502:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3505.965463] LustreError: 216502:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3506.024240] Lustre: server umount lustre-OST0000 complete [ 3513.369381] Lustre: server umount lustre-MDT0000 complete [ 3514.609031] LustreError: 213557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812744 with bad export cookie 8174161980085330672 [ 3514.612069] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3514.614107] LustreError: 213557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3514.754377] Lustre: server umount lustre-MDT0001 complete [ 3517.591892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.753053] LustreError: 217381:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3517.761111] LustreError: 217381:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 7 previous similar messages [ 3519.096903] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3522.037034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3523.358345] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3525.043267] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3526.208414] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3528.412330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3530.399347] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3532.200880] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3533.554461] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3533.639745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3534.966707] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3535.043958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3536.866995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3536.870047] LustreError: Skipped 2 previous similar messages [ 3537.768025] Lustre: server umount lustre-OST0000 complete [ 3539.583515] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3539.654958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3540.882512] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3540.949602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3564.462538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3564.595794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.598649] Lustre: Skipped 5 previous similar messages [ 3566.503899] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3568.255384] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3568.791963] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3575.785743] LustreError: 221788:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3575.787753] LustreError: 221788:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3575.861691] Lustre: server umount lustre-OST0000 complete [ 3578.335868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3578.339116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.342500] Lustre: Skipped 4 previous similar messages [ 3583.246780] Lustre: server umount lustre-MDT0000 complete [ 3584.477119] LustreError: 221994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812814 with bad export cookie 8174161980085331540 [ 3584.479069] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3584.481164] LustreError: 221994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3584.611161] Lustre: server umount lustre-MDT0001 complete [ 3587.476592] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:33:36 (1756812816) [ 3589.655319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.791201] LustreError: 222865:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3591.121165] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3594.037586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3595.509047] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3597.372933] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3598.612374] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3600.811437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3602.753304] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3604.460691] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3605.791599] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3616.075044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3617.362392] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3617.432124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3619.652306] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3619.670341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3619.721347] Lustre: lustre-OST0001: new disk, initializing [ 3619.723983] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3621.526072] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3624.604060] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3625.469153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3625.472098] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3625.505029] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3625.974656] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3626.042352] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3627.322191] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3627.402514] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3630.567230] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3630.568980] Lustre: Skipped 9 previous similar messages [ 3634.110567] Lustre: server umount lustre-OST0001 complete [ 3636.013809] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3636.086416] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3637.411895] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3637.493581] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3656.161687] LustreError: 226645:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.167190] LustreError: 226645:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 14 previous similar messages [ 3661.130600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3663.053606] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3664.762530] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3665.300791] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3671.992975] Lustre: server umount lustre-OST0001 complete [ 3685.855855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3685.859116] LustreError: Skipped 2 previous similar messages [ 3690.078374] Lustre: server umount lustre-OST0000 complete [ 3697.552619] Lustre: server umount lustre-MDT0000 complete [ 3698.782749] LustreError: 228783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812928 with bad export cookie 8174161980085332471 [ 3698.785953] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3698.788446] LustreError: 228783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3704.890757] LustreError: 229413:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3704.892603] LustreError: 229413:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 3704.967556] Lustre: server umount lustre-MDT0001 complete [ 3706.686427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3707.907445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3709.125834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3710.406493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3712.478692] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:35:41 (1756812941) [ 3714.644693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.730539] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3714.741038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3714.809465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.812054] Lustre: Skipped 5 previous similar messages [ 3716.053125] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3718.778477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3718.808851] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3718.819063] Lustre: 230904:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3720.127996] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3721.833104] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3722.957040] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3725.079632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3725.157167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3727.001077] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3728.679151] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3737.345162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3737.371112] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3739.266017] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3740.952767] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3742.542148] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3743.845131] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3743.932498] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3745.229288] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3745.302783] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3746.486339] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff942ae0151000.ost_server_uuid 50 [ 3747.001401] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff942ae0151000.ost_server_uuid in FULL state after 0 sec [ 3747.317233] Lustre: Permanently deactivating lustre-OST0001 [ 3747.322628] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3748.081731] Lustre: Permanently reactivating lustre-OST0001 [ 3753.953215] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.958465] Lustre: Skipped 7 previous similar messages [ 3755.130245] Lustre: server umount lustre-OST0001 complete [ 3762.470919] Lustre: server umount lustre-OST0000 complete [ 3769.813398] Lustre: server umount lustre-MDT0000 complete [ 3770.987607] LustreError: 232543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813000 with bad export cookie 8174161980085333731 [ 3770.992909] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.149610] Lustre: server umount lustre-MDT0001 complete [ 3772.999992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3774.334679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3775.638880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3776.875257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3778.900427] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 07:36:48 (1756813008) [ 3779.225254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3781.351835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.435606] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3781.444829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3781.447797] Lustre: Skipped 5 previous similar messages [ 3782.735588] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3785.434797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3785.465119] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3785.476366] Lustre: 237055:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3786.824840] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3788.614075] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3789.718152] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3791.883579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3791.906037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3791.992359] Lustre: Permanently deactivating lustre-OST0000 [ 3791.996840] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3793.904587] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3795.676247] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3797.960482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3798.215502] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3798.264284] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3799.948671] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3801.726155] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3802.284196] Lustre: Permanently reactivating lustre-OST0000 [ 3807.221771] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3807.224939] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3807.230206] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3807.243988] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3807.250425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3813.861135] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3813.863858] LustreError: Skipped 5 previous similar messages [ 3818.367306] Lustre: server umount lustre-OST0001 complete [ 3818.983231] LustreError: 238681:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3818.988783] LustreError: 238681:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 10 previous similar messages [ 3825.759829] Lustre: server umount lustre-OST0000 complete [ 3827.213308] Lustre: server umount lustre-MDT0000 complete [ 3828.430718] LustreError: 237046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813058 with bad export cookie 8174161980085335348 [ 3828.434539] LustreError: 237046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3828.435160] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3828.602914] Lustre: server umount lustre-MDT0001 complete [ 3831.968335] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3833.064434] Key type lgssc unregistered [ 3833.200400] LNet: 241442:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3833.202828] LNetError: 241442:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3833.215306] LNet: Removed LNI 192.168.202.121@tcp [ 3833.531437] Key type .llcrypt unregistered [ 3833.532416] Key type ._llcrypt unregistered [ 3841.731774] Key type ._llcrypt registered [ 3841.733601] Key type .llcrypt registered [ 3841.769016] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3842.073886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3842.107450] alg: No test for adler32 (adler32-zlib) [ 3842.947022] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 3843.020628] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3844.607163] Key type lgssc registered [ 3844.974559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3846.156775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3848.424428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3848.430572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.578072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3850.858864] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3853.707355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3853.730183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3853.757747] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3853.767767] Lustre: 242860:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3853.773442] Lustre: Permanently deactivating lustre-MDT0001 [ 3853.777926] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3853.872055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3855.139832] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3856.747181] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3856.749628] LustreError: 242902:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3856.752881] LustreError: 242902:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3856.755518] LustreError: 242902:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3856.887243] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3858.159716] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3860.492386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3860.602260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.517694] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3863.714602] LustreError: 244492:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.716748] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3863.720440] LustreError: 244492:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3864.323414] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3866.578393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3866.640042] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3868.482641] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3869.998455] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3870.305202] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3875.113443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3875.113684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3876.094794] Lustre: Permanently reactivating lustre-MDT0001 [ 3876.101729] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3876.104558] Lustre: Skipped 2 previous similar messages [ 3876.135111] Lustre: lustre-MDT0001: Client 6a8136f6-7fcc-4f58-b9d1-1071d55975e6 (at 192.168.202.21@tcp) reconnecting [ 3877.391294] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3877.988051] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3879.824390] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3879.913216] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3881.420968] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3886.639865] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 5 sec [ 3887.536882] Lustre: Permanently deactivating lustre-MDT0001 [ 3887.544610] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3887.546910] Lustre: Skipped 2 previous similar messages [ 3893.250789] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3899.872642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3899.874984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.879859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3901.921571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.921832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3901.926675] Lustre: Skipped 2 previous similar messages [ 3901.928421] Lustre: Skipped 2 previous similar messages [ 3905.854353] LustreError: 246994:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3905.886335] Lustre: server umount lustre-MDT0000 complete [ 3907.040091] LustreError: 243603:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3907.043887] LustreError: 243603:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3907.081187] LustreError: 244510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813136 with bad export cookie 16419782436428968153 [ 3907.084845] LustreError: 244510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3907.085404] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3907.149818] LustreError: 247196:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3907.151784] LustreError: 247196:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3907.222362] Lustre: server umount lustre-MDT0001 complete [ 3914.476192] LustreError: 247397:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3914.478667] LustreError: 247397:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3914.492651] Lustre: server umount lustre-OST0000 complete [ 3918.402973] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 07:39:07 (1756813147) [ 3922.033554] LustreError: 248403:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3922.035613] LustreError: 248403:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3922.118068] Lustre: server umount lustre-OST0001 complete [ 3924.205716] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3927.352365] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3931.089831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3933.289536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3935.369943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3937.563060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3939.895327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3939.929333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.026378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3940.029748] Lustre: Skipped 2 previous similar messages [ 3940.039382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3940.081138] Lustre: lustre-MDT0000: new disk, initializing [ 3940.108702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3940.115738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3941.538286] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3945.653900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3945.682364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3945.716777] Lustre: 250583:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3945.738865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3945.740315] Lustre: Skipped 1 previous similar message [ 3945.776325] Lustre: lustre-MDT0001: new disk, initializing [ 3945.818780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3945.821744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3947.115548] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3950.263461] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3951.455466] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3953.787947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3953.810837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3953.912215] Lustre: lustre-OST0000: new disk, initializing [ 3953.914989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3953.947961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3953.951126] Lustre: Skipped 1 previous similar message [ 3954.999437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3955.002876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3955.045137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3955.882845] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3959.010358] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3959.745666] LustreError: 250590:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3964.008101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3964.032989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3964.097084] Lustre: lustre-OST0001: new disk, initializing [ 3964.099061] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3966.071787] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3969.273866] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3969.830600] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3969.847048] LustreError: 250590:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3969.856830] LustreError: 250590:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3972.604673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3972.609307] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3972.637097] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3979.951123] LustreError: 250590:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3982.817368] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3982.818629] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.820185] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3982.823017] Lustre: Skipped 1 previous similar message [ 3982.824924] Lustre: Skipped 1 previous similar message [ 3987.939702] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3987.942023] Lustre: Skipped 3 previous similar messages [ 3993.062428] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3993.064164] Lustre: Skipped 2 previous similar messages [ 3995.103110] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3995.119001] LustreError: 254097:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3995.120778] LustreError: 254097:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3995.140929] Lustre: server umount lustre-OST0001 complete [ 3998.179645] LustreError: 252938:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.184547] LustreError: 252938:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4003.299312] LustreError: 252938:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4003.303774] LustreError: 252938:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4008.419342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.421740] LustreError: 252316:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4008.423166] Lustre: Skipped 1 previous similar message [ 4008.423704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4008.423710] Lustre: Skipped 1 previous similar message [ 4008.427508] LustreError: 252316:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4013.294122] LustreError: 254321:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4013.297140] LustreError: 254321:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4013.373140] Lustre: server umount lustre-OST0000 complete [ 4017.631991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4017.636314] LustreError: Skipped 1 previous similar message [ 4017.638820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.897150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4028.898901] Lustre: Skipped 7 previous similar messages [ 4029.407136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4029.444717] LustreError: 254523:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4029.447126] LustreError: 254523:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4029.478546] Lustre: server umount lustre-MDT0000 complete [ 4030.784488] LustreError: 251364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813260 with bad export cookie 16419782436428970960 [ 4030.787072] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4030.788525] LustreError: 251364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4045.279091] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4045.417580] Lustre: server umount lustre-MDT0001 complete [ 4049.100725] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4050.240895] Key type lgssc unregistered [ 4050.397431] LNet: 255258:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4050.399664] LNetError: 255258:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4050.408385] LNet: Removed LNI 192.168.202.121@tcp [ 4050.721930] Key type .llcrypt unregistered [ 4050.722945] Key type ._llcrypt unregistered [ 4062.098278] Key type ._llcrypt registered [ 4062.099279] Key type .llcrypt registered [ 4062.135681] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4068.419962] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4068.758791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4068.815527] alg: No test for adler32 (adler32-zlib) [ 4069.702289] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4069.801192] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4071.391154] Key type lgssc registered [ 4071.794466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4074.975600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4077.242791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4079.238181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4081.469266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4083.878483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4083.903640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4083.911428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4085.014617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4085.027345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4085.074648] Lustre: lustre-MDT0000: new disk, initializing [ 4085.102976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4085.112662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4086.538612] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4090.838182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4090.865973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4090.896767] Lustre: 259180:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4090.918106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4090.919690] Lustre: Skipped 1 previous similar message [ 4090.961364] Lustre: lustre-MDT0001: new disk, initializing [ 4090.999472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4091.012929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4091.017473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4092.444829] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4095.740870] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4097.039147] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4099.412313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4099.439780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4099.562919] Lustre: lustre-OST0000: new disk, initializing [ 4099.565774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4099.604339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.410507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4100.413284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4100.459687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4101.623776] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4104.973639] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4106.484926] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4106.567149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4107.952249] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4108.029991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4110.700921] LustreError: 262119:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4110.774317] Lustre: server umount lustre-OST0000 complete [ 4115.937608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.938675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4115.941790] Lustre: Skipped 1 previous similar message [ 4115.946581] Lustre: Skipped 1 previous similar message [ 4121.056169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4121.057679] Lustre: Skipped 1 previous similar message [ 4122.083727] LustreError: 262320:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4122.085657] LustreError: 262320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4122.115491] Lustre: server umount lustre-MDT0000 complete [ 4123.414853] LustreError: 260926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813353 with bad export cookie 4298885993908122931 [ 4123.417279] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4123.420589] LustreError: 260926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4133.859171] LustreError: 262523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4133.861064] LustreError: 262523:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4133.952620] Lustre: server umount lustre-MDT0001 complete [ 4136.950125] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 07:42:46 (1756813366) [ 4139.253151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4139.459439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4140.853679] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4143.809346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4143.950279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4145.393701] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4147.280208] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4148.494641] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4150.868859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4152.993645] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4154.840706] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4205.536530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4205.539188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.543112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4206.564804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.568825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4209.582129] LustreError: 265538:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4209.584151] LustreError: 265538:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4209.668822] Lustre: server umount lustre-OST0000 complete [ 4211.170948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4215.406034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4215.530445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.532570] Lustre: Skipped 1 previous similar message [ 4217.516814] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4219.345735] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4226.786805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4226.791864] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4231.910531] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4231.914311] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4257.764439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4257.765649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.766289] LustreError: Skipped 1 previous similar message [ 4257.766996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4257.769677] Lustre: Skipped 1 previous similar message [ 4257.770915] Lustre: Skipped 2 previous similar messages [ 4258.795890] LustreError: 267204:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4258.799172] LustreError: 267204:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4258.878802] Lustre: server umount lustre-OST0000 complete [ 4261.855904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4261.858234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.862782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4261.864455] Lustre: Skipped 1 previous similar message [ 4266.327591] Lustre: server umount lustre-MDT0000 complete [ 4267.567236] LustreError: 263174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813497 with bad export cookie 4298885993908123799 [ 4267.570809] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4267.573992] LustreError: 263174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4267.649366] LustreError: 267607:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4267.651600] LustreError: 267607:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4267.736156] Lustre: server umount lustre-MDT0001 complete [ 4271.657076] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4272.799606] Key type lgssc unregistered [ 4272.963606] LNet: 268137:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4272.966918] LNetError: 268137:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4272.976527] LNet: Removed LNI 192.168.202.121@tcp [ 4273.351889] Key type .llcrypt unregistered [ 4273.352881] Key type ._llcrypt unregistered [ 4282.506345] Key type ._llcrypt registered [ 4282.507498] Key type .llcrypt registered [ 4282.551265] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4282.933560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4283.002276] alg: No test for adler32 (adler32-zlib) [ 4283.867867] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4283.969270] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4285.559225] Key type lgssc registered [ 4285.988769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4288.709654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4288.719800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.907751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4291.209739] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4294.062401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4294.255122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4295.645216] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4297.399504] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4298.526614] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4300.714402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4300.770265] LustreError: 271111:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4300.773882] LustreError: 271111:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4300.845385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4302.692394] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4303.912595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4304.428970] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4314.081839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.082290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4314.083697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4314.085980] Lustre: Skipped 1 previous similar message [ 4314.092889] LustreError: Skipped 1 previous similar message [ 4318.830751] LustreError: 272528:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4318.900870] Lustre: server umount lustre-OST0000 complete [ 4324.320729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4324.321434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4324.324237] Lustre: Skipped 1 previous similar message [ 4324.326981] Lustre: Skipped 2 previous similar messages [ 4326.332996] LustreError: 272731:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4326.335233] LustreError: 272731:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4326.366198] Lustre: server umount lustre-MDT0000 complete [ 4327.668480] LustreError: 272732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813557 with bad export cookie 17216750161114583783 [ 4327.671584] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4327.671708] LustreError: 272732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4327.741202] LustreError: 272933:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4327.743962] LustreError: 272933:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4327.832804] Lustre: server umount lustre-MDT0001 complete [ 4331.783168] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4332.948321] Key type lgssc unregistered [ 4333.096554] LNet: 273465:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4333.099247] LNetError: 273465:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4333.108420] LNet: Removed LNI 192.168.202.121@tcp [ 4333.449806] Key type .llcrypt unregistered [ 4333.450946] Key type ._llcrypt unregistered [ 4340.560163] Key type ._llcrypt registered [ 4340.561503] Key type .llcrypt registered [ 4340.606822] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4341.043705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4341.076712] alg: No test for adler32 (adler32-zlib) [ 4341.958522] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4342.057634] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4343.647148] Key type lgssc registered [ 4343.998635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4346.746425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4346.753054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4347.918731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4349.260245] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4352.242029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4352.372481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4353.730242] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4355.562856] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4356.829644] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4359.073188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4359.198398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4361.125387] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4362.957435] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4372.515165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4377.570533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4377.571090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.578411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4377.580671] Lustre: Skipped 1 previous similar message [ 4378.990134] LustreError: 277057:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4379.062764] Lustre: server umount lustre-OST0000 complete [ 4382.460057] LustreError: 277258:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4382.462313] LustreError: 277258:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4382.487872] Lustre: server umount lustre-MDT0000 complete [ 4382.689251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.694913] Lustre: Skipped 1 previous similar message [ 4382.697789] LustreError: 275355:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.701847] LustreError: 275355:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4383.758101] LustreError: 274625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813613 with bad export cookie 10315510430785944100 [ 4383.761135] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4383.761791] LustreError: 274625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4383.824722] LustreError: 277460:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4383.826772] LustreError: 277460:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4383.917718] Lustre: server umount lustre-MDT0001 complete [ 4387.690251] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4388.920621] Key type lgssc unregistered [ 4389.082546] LNet: 277991:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4389.085364] LNetError: 277991:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4389.093506] LNet: Removed LNI 192.168.202.121@tcp [ 4389.453641] Key type .llcrypt unregistered [ 4389.454877] Key type ._llcrypt unregistered [ 4399.150329] Key type ._llcrypt registered [ 4399.151907] Key type .llcrypt registered [ 4399.196548] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4399.647786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4399.696563] alg: No test for adler32 (adler32-zlib) [ 4400.571992] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4400.671712] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4402.255198] Key type lgssc registered [ 4402.724562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4405.556735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4405.564837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4406.740563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4408.124878] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4411.128638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4411.258132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4412.746373] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4414.666561] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4415.972350] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4418.299132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4418.436719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4420.369465] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4422.123582] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4432.807577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4446.176036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4446.178239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.183079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4448.226139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4448.226832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.227178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4451.503098] LustreError: 282406:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4451.573869] Lustre: server umount lustre-OST0000 complete [ 4453.345159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.348683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4453.351314] Lustre: Skipped 2 previous similar messages [ 4458.464098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.465763] Lustre: Skipped 1 previous similar message [ 4463.584200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4463.585699] Lustre: Skipped 1 previous similar message [ 4467.167137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4467.213640] LustreError: 282608:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4467.217185] LustreError: 282608:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4467.247939] Lustre: server umount lustre-MDT0000 complete [ 4468.534287] LustreError: 282320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813698 with bad export cookie 1946425769614409768 [ 4468.538549] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4468.540198] LustreError: 282320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4468.608899] LustreError: 282811:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4468.612083] LustreError: 282811:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4468.710149] Lustre: server umount lustre-MDT0001 complete [ 4472.283930] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4473.469455] Key type lgssc unregistered [ 4473.614413] LNet: 283341:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4473.616872] LNetError: 283341:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4473.622462] LNet: Removed LNI 192.168.202.121@tcp [ 4473.968570] Key type .llcrypt unregistered [ 4473.970415] Key type ._llcrypt unregistered [ 4481.421419] Key type ._llcrypt registered [ 4481.422404] Key type .llcrypt registered [ 4481.463239] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4481.855383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4481.865507] alg: No test for adler32 (adler32-zlib) [ 4482.715280] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4482.798301] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4484.383214] Key type lgssc registered [ 4484.784077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4487.511500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4487.518097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4488.705667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.137781] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4493.044715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4493.181559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4494.600267] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4496.402650] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4497.681952] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4500.064750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4500.185930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4502.160330] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4504.012840] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4514.538865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4518.881159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4518.883208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.886542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4519.906354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4519.908359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4519.909285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.916097] Lustre: Skipped 1 previous similar message [ 4520.174115] LustreError: 286974:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4520.249591] Lustre: server umount lustre-OST0000 complete [ 4523.644630] LustreError: 287175:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4523.646513] LustreError: 287175:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4523.673129] Lustre: server umount lustre-MDT0000 complete [ 4524.000142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4524.002831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4524.007337] LustreError: 284548:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4524.011882] LustreError: 284548:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4524.945430] LustreError: 284503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813754 with bad export cookie 8449927734062129939 [ 4524.949749] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4524.950459] LustreError: 284503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4525.016693] LustreError: 287377:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4525.018501] LustreError: 287377:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4525.109430] Lustre: server umount lustre-MDT0001 complete [ 4528.932343] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4530.063483] Key type lgssc unregistered [ 4530.208434] LNet: 287907:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4530.212452] LNetError: 287907:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4530.221412] LNet: Removed LNI 192.168.202.121@tcp [ 4530.562202] Key type .llcrypt unregistered [ 4530.564058] Key type ._llcrypt unregistered [ 4534.369583] Key type ._llcrypt registered [ 4534.370628] Key type .llcrypt registered [ 4534.422829] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4540.137363] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4546.259510] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4546.821134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4546.835542] alg: No test for adler32 (adler32-zlib) [ 4547.694345] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4547.792691] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4549.383227] Key type lgssc registered [ 4549.820116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4552.994916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4555.265291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4557.364804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4559.486622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.871258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.896297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4561.905557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.004585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4563.015637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4563.056346] Lustre: lustre-MDT0000: new disk, initializing [ 4563.081843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4563.088734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4564.443803] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4568.662567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4568.683915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4568.715597] Lustre: 291387:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4568.738731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4568.740363] Lustre: Skipped 1 previous similar message [ 4568.776176] Lustre: lustre-MDT0001: new disk, initializing [ 4568.809955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4568.822778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4568.826204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4570.146702] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4573.294938] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4574.532870] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4576.790402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4576.820033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4576.926083] Lustre: lustre-OST0000: new disk, initializing [ 4576.927803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4576.959721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.925347] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4582.025540] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4583.482115] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4584.957864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4584.961167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4585.004927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4585.614228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4586.924573] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4587.008029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4590.050695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4590.050758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4590.052693] LustreError: Skipped 1 previous similar message [ 4590.053463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4590.055621] Lustre: Skipped 1 previous similar message [ 4590.061067] Lustre: Skipped 1 previous similar message [ 4593.715131] LustreError: 294340:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4593.807460] Lustre: server umount lustre-OST0000 complete [ 4595.169087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.169711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4595.171808] Lustre: Skipped 1 previous similar message [ 4595.175981] Lustre: Skipped 1 previous similar message [ 4600.289677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4600.293037] Lustre: Skipped 1 previous similar message [ 4601.156514] LustreError: 294542:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4601.158662] LustreError: 294542:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4601.196223] Lustre: server umount lustre-MDT0000 complete [ 4602.776220] LustreError: 293132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813832 with bad export cookie 6976767690120313391 [ 4602.779366] LustreError: 293132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4602.779473] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4602.842758] LustreError: 294744:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4602.844262] LustreError: 294744:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4602.923953] Lustre: server umount lustre-MDT0001 complete [ 4606.051968] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 07:50:35 (1756813835) [ 4608.347480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.541265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4609.891443] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4612.666425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4612.782663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4614.346321] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4616.009004] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4617.248383] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4619.782425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4621.945168] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4623.751666] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4638.689289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4638.693069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4638.700192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4639.715287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4643.378110] LustreError: 297682:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4643.380963] LustreError: 297682:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4643.472966] Lustre: server umount lustre-OST0000 complete [ 4647.906898] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4647.909481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.913384] Lustre: Skipped 1 previous similar message [ 4647.915536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4647.920504] Lustre: Skipped 2 previous similar messages [ 4651.397082] LustreError: 297885:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4651.399587] LustreError: 297885:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4651.429472] Lustre: server umount lustre-MDT0000 complete [ 4652.911195] LustreError: 295395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813882 with bad export cookie 6976767690120314273 [ 4652.913689] LustreError: 296135:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4652.913968] LustreError: 295395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4652.914203] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4653.053166] Lustre: server umount lustre-MDT0001 complete [ 4656.811791] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4658.055590] Key type lgssc unregistered [ 4658.197473] LNet: 298617:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4658.200051] LNetError: 298617:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4658.207405] LNet: Removed LNI 192.168.202.121@tcp [ 4658.495458] Key type .llcrypt unregistered [ 4658.496521] Key type ._llcrypt unregistered [ 4668.441937] Key type ._llcrypt registered [ 4668.443690] Key type .llcrypt registered [ 4668.479323] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4675.073353] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4675.389638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4675.416296] alg: No test for adler32 (adler32-zlib) [ 4676.260074] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4676.347404] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4677.927207] Key type lgssc registered [ 4678.296810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4681.479234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4683.644811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4685.778375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4687.987486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4689.732731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4691.577993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4694.036398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4694.060619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4694.065314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4695.160510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4695.172311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4695.212774] Lustre: lustre-MDT0000: new disk, initializing [ 4695.242692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4695.252157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4696.612952] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4701.157098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4701.181784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4701.214503] Lustre: 302394:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4701.244403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4701.247012] Lustre: Skipped 1 previous similar message [ 4701.289969] Lustre: lustre-MDT0001: new disk, initializing [ 4701.329594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4701.344976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4701.348330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4702.669282] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4705.756605] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4707.046279] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4709.430137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4709.452705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4709.558764] Lustre: lustre-OST2710: new disk, initializing [ 4709.560370] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4709.586710] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4711.572716] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4714.688816] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4715.514065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4715.516983] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4715.558977] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4716.996317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4717.027972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4717.088783] Lustre: lustre-OST03e8: new disk, initializing [ 4717.090769] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4717.129337] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4719.078437] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4720.933590] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4721.152244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4721.156246] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4721.189275] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4723.509276] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4723.587015] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4724.908964] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4724.981661] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4726.294609] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4726.369127] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4727.759407] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4727.841131] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4731.366873] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.369994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4731.375479] Lustre: Skipped 2 previous similar messages [ 4731.379071] Lustre: Skipped 1 previous similar message [ 4735.103558] LustreError: 306737:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4735.136275] Lustre: server umount lustre-MDT0000 complete [ 4736.465394] LustreError: 302386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813966 with bad export cookie 5908606305733556899 [ 4736.470145] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4736.470558] LustreError: 302386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4736.481384] Lustre: lustre-MDT0001-lwp-OST2710: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.483521] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4736.486954] Lustre: Skipped 2 previous similar messages [ 4736.488799] Lustre: Skipped 2 previous similar messages [ 4736.531107] LustreError: 306938:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4736.533713] LustreError: 306938:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4736.613467] Lustre: server umount lustre-MDT0001 complete [ 4744.045099] LustreError: 307139:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4744.047518] LustreError: 307139:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4744.070783] Lustre: server umount lustre-OST2710 complete [ 4745.406198] Lustre: server umount lustre-OST03e8 complete [ 4751.424876] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4754.634135] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4757.844890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4760.278449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4762.401163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4764.647115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4767.096681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4767.125319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.231497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4767.242681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4767.283343] Lustre: lustre-MDT0000: new disk, initializing [ 4767.310859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4767.318656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4768.691267] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4773.303871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4773.326212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4773.356594] Lustre: 310129:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4773.475496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4774.815582] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4778.141034] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4779.545381] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4782.046180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4782.077753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4782.175841] Lustre: lustre-OST0000: new disk, initializing [ 4782.178314] Lustre: Skipped 1 previous similar message [ 4782.181564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4782.185105] Lustre: Skipped 2 previous similar messages [ 4782.213255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.214972] Lustre: Skipped 1 previous similar message [ 4784.229168] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4786.687219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4786.689634] Lustre: Skipped 1 previous similar message [ 4786.691913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4786.733842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4787.316915] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4788.600198] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4788.676509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4790.082131] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4790.174501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4791.777196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4791.778502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.780550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4791.782161] Lustre: Skipped 1 previous similar message [ 4791.785609] Lustre: Skipped 1 previous similar message [ 4792.878237] LustreError: 313066:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4792.880255] LustreError: 313066:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4792.946709] Lustre: server umount lustre-OST0000 complete [ 4796.896861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.897910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4796.902235] Lustre: Skipped 1 previous similar message [ 4800.321606] LustreError: 313266:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4800.324051] LustreError: 313266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4800.357010] Lustre: server umount lustre-MDT0000 complete [ 4801.669487] LustreError: 310165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814031 with bad export cookie 5908606305733558579 [ 4801.674370] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4801.675735] LustreError: 310165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4801.826641] Lustre: server umount lustre-MDT0001 complete [ 4804.707848] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:53:53 (1756814033) [ 4805.188369] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4805.774970] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:53:54 (1756814034) [ 4806.746065] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4809.643445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4810.980842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4812.243282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4813.573870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4813.949334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4816.265518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.366164] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4816.373226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4816.436449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.803773] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4820.771614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4820.804630] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4820.814373] Lustre: 315591:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4822.253350] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4824.082632] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4825.259366] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4827.490750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4827.519360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4827.596456] Lustre: Denying initial registration attempt from nid 192.168.202.121@tcp, specified as failover [ 4827.598986] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4827.602182] LustreError: 317209:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -99 [ 4827.604115] LustreError: 317209:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 4827.605560] LustreError: 317209:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4827.660803] Lustre: server umount lustre-OST0000 complete [ 4827.662044] LustreError: 317209:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4831.199859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4831.202041] LustreError: Skipped 1 previous similar message [ 4831.203305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.207585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4831.209481] Lustre: Skipped 1 previous similar message [ 4835.966058] LustreError: 317576:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4835.967740] LustreError: 317576:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4835.996981] Lustre: server umount lustre-MDT0000 complete [ 4837.210252] LustreError: 315583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814066 with bad export cookie 5908606305733559447 [ 4837.215866] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4837.361830] Lustre: server umount lustre-MDT0001 complete [ 4840.928925] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4842.094701] Key type lgssc unregistered [ 4842.240318] LNet: 318310:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4842.242723] LNetError: 318310:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4842.253409] LNet: Removed LNI 192.168.202.121@tcp [ 4842.588309] Key type .llcrypt unregistered [ 4842.589316] Key type ._llcrypt unregistered [ 4846.210057] Key type ._llcrypt registered [ 4846.210924] Key type .llcrypt registered [ 4846.251342] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4846.679995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4846.726904] alg: No test for adler32 (adler32-zlib) [ 4847.584249] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4847.668957] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4849.247198] Key type lgssc registered [ 4849.697879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4852.165109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4853.433620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4854.754425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4856.069492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4856.440919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4863.094911] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4866.099951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4866.107844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.215438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4867.222335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4867.288637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4868.565182] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4871.502772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4871.534994] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4871.548198] Lustre: 321043:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4871.649357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4873.224365] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4875.274748] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4876.560401] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4879.101503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4879.128439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4879.201234] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4879.266087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.150820] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4883.127683] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4885.474103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4885.474571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.475928] LustreError: Skipped 1 previous similar message [ 4885.481838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4885.484823] Lustre: Skipped 1 previous similar message [ 4886.061738] LustreError: 323326:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4886.124986] Lustre: server umount lustre-OST0000 complete [ 4890.592530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.593377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4890.599059] Lustre: Skipped 2 previous similar messages [ 4893.500511] LustreError: 323526:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4893.502341] LustreError: 323526:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4893.530884] Lustre: server umount lustre-MDT0000 complete [ 4894.831445] LustreError: 321078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814124 with bad export cookie 1578730347202201691 [ 4894.834751] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4894.836261] LustreError: 321078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4894.899322] LustreError: 323729:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4894.900917] LustreError: 323729:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4894.988796] Lustre: server umount lustre-MDT0001 complete [ 4899.021479] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4900.299783] Key type lgssc unregistered [ 4900.456428] LNet: 324260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4900.460539] LNetError: 324260:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4900.467429] LNet: Removed LNI 192.168.202.121@tcp [ 4900.811410] Key type .llcrypt unregistered [ 4900.812437] Key type ._llcrypt unregistered [ 4910.279179] Key type ._llcrypt registered [ 4910.280659] Key type .llcrypt registered [ 4910.321653] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4910.731462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4910.774589] alg: No test for adler32 (adler32-zlib) [ 4911.642485] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4911.741223] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4913.327204] Key type lgssc registered [ 4913.741210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4916.992116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4916.998793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4918.164858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.508951] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4923.030896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4923.191122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4924.948093] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4926.999019] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4928.466266] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4931.001428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4931.129536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4933.124675] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4935.023040] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4948.450157] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.451138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4948.453048] Lustre: Skipped 2 previous similar messages [ 4948.454531] Lustre: Skipped 1 previous similar message [ 4950.848669] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 4950.850468] Lustre: Skipped 1 previous similar message [ 4953.568996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4953.572290] Lustre: Skipped 2 previous similar messages [ 4953.860810] LustreError: 327902:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4953.889645] Lustre: server umount lustre-MDT0000 complete [ 4955.229435] LustreError: 326352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814184 with bad export cookie 3345464709716082132 [ 4955.231389] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4955.232463] LustreError: 326352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4955.962558] LustreError: 325628:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4955.967021] LustreError: 325628:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4955.969500] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) [ 4958.687982] LustreError: 327789:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.688286] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4960.736084] LustreError: 325628:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4960.736200] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4960.745634] Lustre: Skipped 1 previous similar message [ 4961.341969] LustreError: 328107:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4961.345609] LustreError: 328107:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4961.435719] Lustre: server umount lustre-MDT0001 complete [ 4963.801439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4966.981756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4980.191777] LustreError: 324944:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000003735c60 x1842153144737152/t0(0) o250->MGC192.168.202.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4980.320630] LustreError: 327241:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4980.326519] LustreError: 327241:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4980.348242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4980.372245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4981.645495] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4984.483179] LustreError: 328916:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4984.487752] LustreError: 328916:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4985.104166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4985.259220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4986.686296] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4988.718370] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4990.373398] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4991.339128] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4992.538027] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4993.047660] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4995.556813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4995.559908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.563245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4995.571684] Lustre: Skipped 1 previous similar message [ 4995.578192] Lustre: Skipped 1 previous similar message [ 5000.366193] LustreError: 330409:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5000.368125] LustreError: 330409:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5000.437765] Lustre: server umount lustre-OST0000 complete [ 5005.794170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5005.794764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.806282] Lustre: Skipped 1 previous similar message [ 5008.016776] LustreError: 330610:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5008.020394] LustreError: 330610:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5008.055996] Lustre: server umount lustre-MDT0000 complete [ 5009.361634] LustreError: 326352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814238 with bad export cookie 3345464709716089286 [ 5009.364837] LustreError: 326352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5009.365576] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5009.511735] Lustre: server umount lustre-MDT0001 complete [ 5013.213355] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5014.394246] Key type lgssc unregistered [ 5014.523391] LNet: 331342:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5014.525603] LNetError: 331342:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5014.535274] LNet: Removed LNI 192.168.202.121@tcp [ 5014.870615] Key type .llcrypt unregistered [ 5014.872891] Key type ._llcrypt unregistered [ 5024.524546] Key type ._llcrypt registered [ 5024.525425] Key type .llcrypt registered [ 5024.566259] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5024.940732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5025.012839] alg: No test for adler32 (adler32-zlib) [ 5025.869936] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 5025.955586] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5027.535203] Key type lgssc registered [ 5028.020359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5030.637757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5030.643424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.816340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.093146] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5036.441468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5036.586647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5037.885289] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5039.765317] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5040.984539] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5046.754525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.754656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5046.755931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.763267] Lustre: Skipped 1 previous similar message [ 5047.840951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5047.843562] Lustre: Skipped 2 previous similar messages [ 5048.133799] LustreError: 334106:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5048.160562] Lustre: server umount lustre-MDT0000 complete [ 5049.598405] LustreError: 334107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814279 with bad export cookie 5808356969045549869 [ 5049.602915] LustreError: 334107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5049.603566] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5049.685427] LustreError: 334309:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5049.687411] LustreError: 334309:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5049.795457] Lustre: server umount lustre-MDT0001 complete [ 5054.854655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.946658] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5054.956985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5055.031329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5056.453849] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5059.435081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5059.467434] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5059.479355] Lustre: 335075:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5059.583883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5060.973135] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5062.771205] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5063.999127] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5066.843797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5066.915607] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5066.918493] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5066.921452] LustreError: 336731:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -2 [ 5066.923282] LustreError: 336731:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 5066.924888] LustreError: 336731:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5066.974775] Lustre: server umount lustre-OST0000 complete [ 5066.976381] LustreError: 336731:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5069.751238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5069.816225] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5069.886453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5071.823274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5071.949255] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5073.762403] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5076.646028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5076.679285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5076.706446] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5076.743656] Lustre: lustre-OST0001: new disk, initializing [ 5076.745249] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5078.268750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5078.271439] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5078.309041] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5078.776822] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5080.643721] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5083.620960] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5083.621565] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.623684] LustreError: Skipped 1 previous similar message [ 5083.628547] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5083.631661] Lustre: Skipped 1 previous similar message [ 5087.658223] LustreError: 338848:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5087.660067] LustreError: 338848:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5087.681501] Lustre: server umount lustre-OST0001 complete [ 5088.741743] LustreError: 338851:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5088.747433] LustreError: 338851:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5092.320650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5092.320853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.323681] LustreError: Skipped 1 previous similar message [ 5092.328360] Lustre: Skipped 1 previous similar message [ 5092.330676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5092.333289] Lustre: Skipped 1 previous similar message [ 5093.859333] LustreError: 337200:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5093.863055] LustreError: 337200:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5095.027138] LustreError: 339053:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5095.030513] LustreError: 339053:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5095.116343] Lustre: server umount lustre-OST0000 complete [ 5098.978217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5098.978991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5098.981484] Lustre: Skipped 1 previous similar message [ 5098.987016] Lustre: Skipped 3 previous similar messages [ 5109.216047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5109.217738] Lustre: Skipped 3 previous similar messages [ 5111.263164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5111.295953] LustreError: 339254:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5111.297959] LustreError: 339254:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5111.332210] Lustre: server umount lustre-MDT0000 complete [ 5112.697917] LustreError: 335068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814342 with bad export cookie 5808356969045550408 [ 5112.703325] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5112.703670] LustreError: 335068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5112.892606] Lustre: server umount lustre-MDT0001 complete [ 5116.240139] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5117.223536] Key type lgssc unregistered [ 5117.350310] LNet: 339991:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5117.352250] LNetError: 339991:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5117.361302] LNet: Removed LNI 192.168.202.121@tcp [ 5117.753866] Key type .llcrypt unregistered [ 5117.755199] Key type ._llcrypt unregistered [ 5131.003148] Key type ._llcrypt registered [ 5131.004678] Key type .llcrypt registered [ 5131.066997] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 5137.538457] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5137.968438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5137.974800] alg: No test for adler32 (adler32-zlib) [ 5138.834836] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 5138.922197] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5140.503197] Key type lgssc registered [ 5140.976113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5144.672246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5147.385691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5149.504953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5152.365267] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5154.789238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5154.814560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5154.820427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.933047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5155.947596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5155.988972] Lustre: lustre-MDT0000: new disk, initializing [ 5156.023253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.030466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5157.410489] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5162.204574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5162.230211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5162.266936] Lustre: 343914:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5162.291695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5162.294167] Lustre: Skipped 1 previous similar message [ 5162.336379] Lustre: lustre-MDT0001: new disk, initializing [ 5162.375572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5162.390124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5162.394587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5163.705414] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5166.878840] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5168.180615] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5170.502597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5170.529510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5170.633576] Lustre: lustre-OST0000: new disk, initializing [ 5170.635851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5170.669311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5172.528983] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5173.691204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5173.694705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5173.729619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5175.496834] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5176.774708] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5176.858375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5178.117823] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5178.189824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5178.848841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5178.849953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5178.850870] LustreError: Skipped 1 previous similar message [ 5178.851408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5178.855778] Lustre: Skipped 1 previous similar message [ 5183.970666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5183.972230] Lustre: Skipped 2 previous similar messages [ 5184.876094] LustreError: 346854:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5184.937850] Lustre: server umount lustre-OST0000 complete [ 5188.063978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5188.067604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5188.074365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5189.089992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5192.253247] LustreError: 347055:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5192.254907] LustreError: 347055:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5192.288706] Lustre: server umount lustre-MDT0000 complete [ 5193.483557] LustreError: 344696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814423 with bad export cookie 540924639115523167 [ 5193.488583] LustreError: 344696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5193.488605] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5193.551926] LustreError: 347256:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5193.553560] LustreError: 347256:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5193.648499] Lustre: server umount lustre-MDT0001 complete [ 5196.231773] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 08:00:25 (1756814425) [ 5198.210149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5200.109596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5207.724242] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 5210.867557] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5214.220730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5216.355837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5218.296389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5220.387902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5222.803895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5222.823187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.916158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5222.926481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5222.966566] Lustre: lustre-MDT0000: new disk, initializing [ 5222.991207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.997466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5224.332547] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5228.504463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5228.522919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5228.556433] Lustre: 351022:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5228.587726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5228.589176] Lustre: Skipped 1 previous similar message [ 5228.628029] Lustre: lustre-MDT0001: new disk, initializing [ 5228.668615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5228.684418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5228.688682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5229.921634] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5233.219641] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5234.514405] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5236.844232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5236.860692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5236.957188] Lustre: lustre-OST0000: new disk, initializing [ 5236.958744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5236.986504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5238.894286] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5242.239667] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5243.619330] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5245.437476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5245.442832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5245.482790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5245.752748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5247.067461] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5247.141695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5250.528724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5250.529212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.530760] LustreError: Skipped 1 previous similar message [ 5250.531780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5250.533925] Lustre: Skipped 1 previous similar message [ 5250.539412] Lustre: Skipped 3 previous similar messages [ 5253.870170] LustreError: 353986:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5253.872646] LustreError: 353986:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5253.946568] Lustre: server umount lustre-OST0000 complete [ 5255.648521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.649151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5255.651653] Lustre: Skipped 1 previous similar message [ 5261.377765] LustreError: 354188:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5261.379757] LustreError: 354188:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5261.411153] Lustre: server umount lustre-MDT0000 complete [ 5262.672352] LustreError: 351802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814492 with bad export cookie 540924639115524035 [ 5262.675633] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5262.676066] LustreError: 351802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5262.840379] Lustre: server umount lustre-MDT0001 complete [ 5265.603772] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:01:34 (1756814494) [ 5268.006990] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:01:37 (1756814497) [ 5270.793868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.988293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5272.395302] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5275.244983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5276.662031] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5278.403429] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5279.601680] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5281.906553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5283.953117] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5285.831331] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5299.848168] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5300.397448] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5300.953479] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5301.532508] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5302.176106] Lustre: Failing over lustre-MDT0000 [ 5302.359884] LustreError: 357861:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5302.361970] LustreError: 357861:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5302.391143] Lustre: server umount lustre-MDT0000 complete [ 5306.336182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5306.338337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5306.342826] LustreError: 355402:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.346444] LustreError: 355402:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 5306.852871] LustreError: 355398:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.857056] LustreError: 355398:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 5311.803817] LustreError: 356765:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.301913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.341319] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5315.427609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5315.431114] Lustre: Skipped 2 previous similar messages [ 5315.454689] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5316.737837] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5316.924085] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5320.678221] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5320.684874] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5320.702952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5321.841856] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5322.361624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5324.180263] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5324.821669] Lustre: Failing over lustre-MDT0000 [ 5324.968522] LustreError: 358938:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5324.970479] LustreError: 358938:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5325.003260] Lustre: server umount lustre-MDT0000 complete [ 5325.794238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5325.794951] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5325.796503] LustreError: 355421:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.796513] LustreError: 355421:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5325.805493] Lustre: Skipped 4 previous similar messages [ 5330.912829] LustreError: 355421:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.916817] LustreError: 355421:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 5337.979820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5338.025467] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5338.183125] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5339.888555] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5340.487815] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5343.205317] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5343.206919] Lustre: Skipped 2 previous similar messages [ 5343.212737] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5343.228696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5344.339952] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5344.910564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5348.321718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5348.323346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5348.325675] Lustre: Skipped 5 previous similar messages [ 5353.302123] Lustre: server umount lustre-MDT0000 complete [ 5353.440626] LustreError: 355403:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5353.446116] LustreError: 355403:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 7 previous similar messages [ 5354.492315] LustreError: 355383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814584 with bad export cookie 540924639115526674 [ 5354.495837] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5354.495841] LustreError: 355383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5354.644630] Lustre: server umount lustre-MDT0001 complete [ 5366.195597] LustreError: 360370:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5366.197485] LustreError: 360370:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5366.285462] Lustre: server umount lustre-OST0000 complete [ 5371.635244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.798244] LustreError: 361199:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.136390] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5376.151556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5377.614624] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5379.390621] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5380.571113] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5382.910532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5383.016234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.018746] Lustre: Skipped 3 previous similar messages [ 5384.902802] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5386.620498] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5397.351092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5406.688470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5406.692377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5406.697752] Lustre: Skipped 3 previous similar messages [ 5406.700716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5409.777414] Lustre: server umount lustre-OST0000 complete [ 5417.265443] Lustre: server umount lustre-MDT0000 complete [ 5417.953310] LustreError: 361217:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.959352] LustreError: 361217:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5418.648072] LustreError: 361180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814648 with bad export cookie 540924639115527402 [ 5418.650813] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5418.651614] LustreError: 361180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5418.807555] Lustre: server umount lustre-MDT0001 complete [ 5422.701525] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5423.798258] Key type lgssc unregistered [ 5423.928359] LNet: 364398:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5423.930553] LNetError: 364398:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5423.937316] LNet: Removed LNI 192.168.202.121@tcp [ 5424.242043] Key type .llcrypt unregistered [ 5424.243158] Key type ._llcrypt unregistered [ 5434.051878] Key type ._llcrypt registered [ 5434.053545] Key type .llcrypt registered [ 5434.097900] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5434.406922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5434.462306] alg: No test for adler32 (adler32-zlib) [ 5435.329507] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 5435.425188] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5437.015107] Key type lgssc registered [ 5437.369729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5439.793098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5439.799482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5440.960461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5442.271285] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5445.077000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5445.192382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5446.537909] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5448.287929] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5449.447929] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5451.591665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5451.702965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.443968] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5455.080177] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5466.022335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5475.807804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5475.809762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5475.813658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5476.321104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5476.323759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.326589] Lustre: Skipped 1 previous similar message [ 5480.573216] LustreError: 368140:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5480.601698] Lustre: server umount lustre-MDT0000 complete [ 5481.440713] LustreError: 365870:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5481.444630] LustreError: 365870:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 5481.825255] LustreError: 365852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814711 with bad export cookie 1550601963582928883 [ 5481.828726] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.831757] LustreError: 365852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5481.901623] LustreError: 368343:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5481.904282] LustreError: 368343:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5481.981625] Lustre: server umount lustre-MDT0001 complete [ 5492.402099] LustreError: 368543:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5492.404185] LustreError: 368543:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5492.476336] Lustre: server umount lustre-OST0000 complete [ 5497.009760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5497.165055] LustreError: 369271:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5497.191252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5498.545607] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5501.832093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5501.964229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5503.330945] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5505.133327] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5506.481805] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5508.929730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5510.906166] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5512.738174] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5523.365680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5537.760081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5537.762093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5537.765234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.766619] Lustre: Skipped 2 previous similar messages [ 5541.319120] LustreError: 371640:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5541.322598] LustreError: 371640:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5541.353957] Lustre: server umount lustre-MDT0000 complete [ 5542.646094] LustreError: 369253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814772 with bad export cookie 1550601963582929989 [ 5542.651256] LustreError: 369253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5542.653879] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5542.789528] Lustre: server umount lustre-MDT0001 complete [ 5554.608635] LustreError: 372044:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5554.610405] LustreError: 372044:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5554.688434] Lustre: server umount lustre-OST0000 complete [ 5559.783260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5559.941530] LustreError: 372871:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5559.967750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.971243] Lustre: Skipped 1 previous similar message [ 5561.315973] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5564.242028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5565.634411] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5567.332704] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5568.443453] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5570.536407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5572.467561] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5574.264221] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5584.995321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5594.592146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5594.594250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.597306] Lustre: Skipped 2 previous similar messages [ 5594.598905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5594.600339] Lustre: Skipped 3 previous similar messages [ 5600.737054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5600.738757] Lustre: Skipped 3 previous similar messages [ 5605.343106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5605.367984] LustreError: 375138:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5605.373156] LustreError: 375138:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5605.466492] Lustre: server umount lustre-OST0000 complete [ 5610.464121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5610.467747] LustreError: Skipped 1 previous similar message [ 5610.469825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.475410] Lustre: Skipped 1 previous similar message [ 5610.477949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5621.215183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5621.277551] Lustre: server umount lustre-MDT0000 complete [ 5622.442366] LustreError: 372851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814852 with bad export cookie 1550601963582930997 [ 5622.445070] LustreError: 372851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5622.445190] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5622.508765] LustreError: 375546:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5622.510430] LustreError: 375546:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5622.592723] Lustre: server umount lustre-MDT0001 complete [ 5625.965807] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5627.017085] Key type lgssc unregistered [ 5627.186669] LNet: 376078:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5627.189852] LNetError: 376078:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5627.203581] LNet: Removed LNI 192.168.202.121@tcp [ 5627.690593] Key type .llcrypt unregistered [ 5627.692429] Key type ._llcrypt unregistered [ 5636.804662] Key type ._llcrypt registered [ 5636.805819] Key type .llcrypt registered [ 5636.846726] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5637.282342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5637.295167] alg: No test for adler32 (adler32-zlib) [ 5638.150260] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 5638.235617] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5639.815230] Key type lgssc registered [ 5640.184667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5642.795639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5642.800736] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5642.802807] LustreError: 377477:0:(osd_handler.c:8487:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5642.809968] LustreError: 377477:0:(obd_config.c:782:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5642.813166] LustreError: 377477:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5642.817397] LustreError: 377477:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5642.821137] LustreError: 377477:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5645.373369] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5645.378611] LustreError: 377874:0:(osd_handler.c:8487:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5645.387981] LustreError: 377874:0:(obd_config.c:782:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5645.390022] LustreError: 377874:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5645.392051] LustreError: 377874:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5645.394057] LustreError: 377874:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5651.791949] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5652.919252] Key type lgssc unregistered [ 5653.064314] LNet: 378857:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5653.066985] LNetError: 378857:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5653.072363] LNet: Removed LNI 192.168.202.121@tcp [ 5653.406459] Key type .llcrypt unregistered [ 5653.407715] Key type ._llcrypt unregistered [ 5659.298049] Key type ._llcrypt registered [ 5659.299298] Key type .llcrypt registered [ 5659.335163] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 5665.582630] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5665.929970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5665.990082] alg: No test for adler32 (adler32-zlib) [ 5666.840269] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 5666.918976] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5668.495155] Key type lgssc registered [ 5668.850337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5671.890463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5674.079783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5676.044247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5678.115417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5680.366547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5680.394910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5680.401694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.503392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5681.517890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5681.556917] Lustre: lustre-MDT0000: new disk, initializing [ 5681.583217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.590849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5682.907206] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5687.126051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5687.153306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5687.184232] Lustre: 381961:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5687.205864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5687.207401] Lustre: Skipped 1 previous similar message [ 5687.241018] Lustre: lustre-MDT0001: new disk, initializing [ 5687.272224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5687.284758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5687.287605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5688.542937] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5691.492411] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5692.584317] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5694.844409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5694.868311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5694.955997] Lustre: lustre-OST0000: new disk, initializing [ 5694.958916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5694.986959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.848379] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5699.976117] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5701.114239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5701.119207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5701.156676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5701.293349] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5701.368535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5702.641461] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5702.714687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5706.208983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5706.209497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.210832] LustreError: Skipped 1 previous similar message [ 5706.215138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.217426] Lustre: Skipped 1 previous similar message [ 5709.421775] LustreError: 384900:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5709.487158] Lustre: server umount lustre-OST0000 complete [ 5711.328744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.329597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5711.331480] Lustre: Skipped 2 previous similar messages [ 5711.335059] Lustre: Skipped 1 previous similar message [ 5716.448149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5716.449549] Lustre: Skipped 1 previous similar message [ 5716.794682] LustreError: 385102:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5716.796416] LustreError: 385102:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5716.822954] Lustre: server umount lustre-MDT0000 complete [ 5717.974746] LustreError: 381952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814947 with bad export cookie 1551444814361554901 [ 5717.977691] LustreError: 381952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5717.978469] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5718.032489] LustreError: 385303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5718.033985] LustreError: 385303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5718.111819] Lustre: server umount lustre-MDT0001 complete [ 5720.741484] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5721.261410] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:09:10 (1756814950) [ 5723.504575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5723.672687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.894784] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5727.616538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5727.719757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5728.956385] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5730.567115] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5732.050404] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5734.595624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5737.117143] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5739.164535] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5741.421239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5741.453101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5741.505782] Lustre: lustre-OST0001: new disk, initializing [ 5741.508751] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5741.536520] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5741.538256] Lustre: Skipped 1 previous similar message [ 5743.812884] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5744.514213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5744.516825] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5744.549550] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5747.339653] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5749.728272] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5749.728406] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.730444] LustreError: Skipped 1 previous similar message [ 5749.733746] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5749.735878] Lustre: Skipped 1 previous similar message [ 5753.828589] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5753.830629] Lustre: Skipped 2 previous similar messages [ 5755.373533] LustreError: 389221:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5755.376674] LustreError: 389221:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5755.394488] Lustre: server umount lustre-OST0001 complete [ 5758.946159] LustreError: 387616:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.950563] LustreError: 387616:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5764.069441] LustreError: 388048:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5764.076820] LustreError: 388048:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5768.672580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5768.676506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5768.681488] Lustre: Skipped 1 previous similar message [ 5768.683137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5769.187719] LustreError: 388048:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.192027] LustreError: 388048:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5773.546833] LustreError: 389445:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5773.549310] LustreError: 389445:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5773.610729] Lustre: server umount lustre-OST0000 complete [ 5778.911966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5778.914655] LustreError: Skipped 1 previous similar message [ 5778.916225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.921375] Lustre: Skipped 1 previous similar message [ 5781.073623] Lustre: server umount lustre-MDT0000 complete [ 5782.260276] LustreError: 386000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815011 with bad export cookie 1551444814361555755 [ 5782.264470] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.412379] LustreError: 389848:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5788.414500] LustreError: 389848:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5788.496642] Lustre: server umount lustre-MDT0001 complete [ 5792.265264] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5793.302649] Key type lgssc unregistered [ 5793.434310] LNet: 390381:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5793.436770] LNetError: 390381:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5793.442262] LNet: Removed LNI 192.168.202.121@tcp [ 5793.739210] Key type .llcrypt unregistered [ 5793.740759] Key type ._llcrypt unregistered [ 5804.908223] Key type ._llcrypt registered [ 5804.909402] Key type .llcrypt registered [ 5804.949340] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 5811.205522] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5811.570835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5811.582674] alg: No test for adler32 (adler32-zlib) [ 5812.461980] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 5812.555636] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5814.143255] Key type lgssc registered [ 5814.565741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5817.696712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5820.042652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5821.967518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5824.136333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5826.399147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5826.423419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5826.428290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5827.527466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5827.538741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5827.573593] Lustre: lustre-MDT0000: new disk, initializing [ 5827.601239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.608952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5828.909818] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5833.140373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5833.164674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5833.198351] Lustre: 394301:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5833.222155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5833.223675] Lustre: Skipped 1 previous similar message [ 5833.264268] Lustre: lustre-MDT0001: new disk, initializing [ 5833.299314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5833.314398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5833.319040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5834.723197] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5837.877266] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5839.055447] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5841.344826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5841.365291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5841.489729] Lustre: lustre-OST0000: new disk, initializing [ 5841.491502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5841.528979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.009845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5843.014996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5843.056395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5843.535516] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5846.539938] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5847.906807] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5848.001075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5849.380794] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5849.469305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5853.152120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5853.155713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5853.162086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5853.668800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5853.669300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5853.675187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5856.237065] LustreError: 397240:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5856.314186] Lustre: server umount lustre-OST0000 complete [ 5858.784797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5858.785268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.786285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5858.791838] Lustre: Skipped 1 previous similar message [ 5863.677140] LustreError: 397441:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5863.678779] LustreError: 397441:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5863.708440] Lustre: server umount lustre-MDT0000 complete [ 5864.947223] LustreError: 395082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815094 with bad export cookie 12689883704882029503 [ 5864.949623] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5864.949870] LustreError: 395082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5865.018614] LustreError: 397644:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5865.021323] LustreError: 397644:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5865.114670] Lustre: server umount lustre-MDT0001 complete [ 5867.916734] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 08:11:37 (1756815097) [ 5870.849532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5873.579130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5873.719434] Lustre: lustre-MDT0000: new disk, initializing [ 5873.731039] LustreError: 398910:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.736935] LustreError: 398910:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5873.752216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.027089] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5877.882037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5878.032682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5879.452546] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5881.315237] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5882.605245] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5883.319504] LustreError: 398938:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5883.322354] LustreError: 398938:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5883.324609] LustreError: 398938:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 5883.360273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.361484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5883.361583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5883.366667] Lustre: Skipped 1 previous similar message [ 5883.371729] Lustre: Skipped 2 previous similar messages [ 5887.456374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5887.458708] Lustre: Skipped 1 previous similar message [ 5889.469299] LustreError: 400250:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5889.471137] LustreError: 400250:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5889.494950] Lustre: server umount lustre-MDT0000 complete [ 5890.765258] LustreError: 398891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815120 with bad export cookie 12689883704882030371 [ 5890.768483] LustreError: 398891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5890.768667] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5890.925685] Lustre: server umount lustre-MDT0001 complete [ 5893.966542] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:12:03 (1756815123) [ 5896.221138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5896.389058] LustreError: 401172:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5896.416778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.786085] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5900.682766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5902.120019] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5903.921841] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5905.210848] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5907.422276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5909.481236] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5911.318978] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5929.732895] Lustre: Permanently deactivating lustre-OST0000 [ 5929.741305] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5938.159957] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5939.775695] LustreError: 403981:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5939.778424] LustreError: 403981:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5939.780207] LustreError: 403981:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5940.374603] LustreError: 404029:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5940.376654] LustreError: 404029:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5947.052076] LustreError: 404126:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5947.053673] LustreError: 404126:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5947.107698] Lustre: server umount lustre-OST0000 complete [ 5947.961307] LustreError: 404278:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5947.965092] LustreError: 404278:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5951.967786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5951.969897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.973743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5959.138252] LustreError: 404376:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5959.139714] LustreError: 404376:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5959.159636] Lustre: server umount lustre-MDT0000 complete [ 5960.332263] LustreError: 401154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815189 with bad export cookie 12689883704882030903 [ 5960.334505] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5960.335124] LustreError: 401154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5969.959770] Lustre: server umount lustre-MDT0001 complete [ 5972.807153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.163612] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5975.119500] LustreError: 405421:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5977.140670] Lustre: 405759:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.121@tcp [ 5977.787407] LustreError: 405856:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5977.789415] LustreError: 405856:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5977.811527] Lustre: server umount MGS complete [ 5981.494529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5981.642481] LustreError: 406484:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.647579] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5981.653133] Lustre: Skipped 1 previous similar message [ 5981.673656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5981.677372] Lustre: Skipped 2 previous similar messages [ 5982.900770] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5985.737381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5985.837077] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5987.171619] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5988.920843] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5990.069926] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5990.696176] Lustre: Permanently reactivating lustre-OST0000 [ 5990.700450] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5990.702357] Lustre: Skipped 2 previous similar messages [ 6013.919065] LustreError: 407963:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6016.273398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6016.401469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6016.403154] Lustre: Skipped 1 previous similar message [ 6018.303421] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6020.071128] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6033.890518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.890813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6033.893986] Lustre: Skipped 1 previous similar message [ 6033.897981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6033.899906] Lustre: Skipped 5 previous similar messages [ 6037.994078] LustreError: 409769:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6037.995657] LustreError: 409769:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6038.054615] Lustre: server umount lustre-OST0000 complete [ 6045.396241] Lustre: server umount lustre-MDT0000 complete [ 6046.597149] LustreError: 407202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815276 with bad export cookie 12689883704882031848 [ 6046.600160] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6046.604196] LustreError: 407202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6046.741668] Lustre: server umount lustre-MDT0001 complete [ 6050.244730] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6051.404234] Key type lgssc unregistered [ 6051.549423] LNet: 410707:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6051.551615] LNetError: 410707:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6051.559467] LNet: Removed LNI 192.168.202.121@tcp [ 6051.945632] Key type .llcrypt unregistered [ 6051.946691] Key type ._llcrypt unregistered [ 6058.402377] Key type ._llcrypt registered [ 6058.403374] Key type .llcrypt registered [ 6058.443209] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 6065.017313] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6065.542845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6065.556463] alg: No test for adler32 (adler32-zlib) [ 6066.419767] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 6066.512073] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6068.095182] Key type lgssc registered [ 6068.503218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6071.493235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6073.640302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6075.511085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6077.634095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6079.984107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6080.006313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6080.013621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6081.099853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6081.110177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6081.148113] Lustre: lustre-MDT0000: new disk, initializing [ 6081.172968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6081.179132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6082.455418] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6086.524164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6086.551908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6086.590661] Lustre: 413812:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6086.611705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6086.613284] Lustre: Skipped 1 previous similar message [ 6086.646793] Lustre: lustre-MDT0001: new disk, initializing [ 6086.676908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6086.688628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6086.692388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6087.936734] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6090.882199] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6092.007119] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6094.258853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6094.283712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6094.407245] Lustre: lustre-OST0000: new disk, initializing [ 6094.409541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6094.443675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.314297] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6097.277554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6097.280490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6097.325379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6099.403650] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6100.824912] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6100.910449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6102.274664] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6102.358803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6107.617056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6107.618439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6107.619918] LustreError: Skipped 1 previous similar message [ 6107.627076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6107.631219] Lustre: Skipped 1 previous similar message [ 6112.737791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6112.739810] Lustre: Skipped 1 previous similar message [ 6117.343094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6117.360095] LustreError: 416747:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6117.417623] Lustre: server umount lustre-OST0000 complete [ 6122.463939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6122.467297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.472610] Lustre: Skipped 1 previous similar message [ 6122.475091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6124.732433] LustreError: 416951:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6124.734191] LustreError: 416951:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6124.757723] Lustre: server umount lustre-MDT0000 complete [ 6125.900125] LustreError: 414591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815355 with bad export cookie 14716897217338950773 [ 6125.902146] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6125.903332] LustreError: 414591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6125.964558] LustreError: 417153:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6125.966489] LustreError: 417153:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6126.042785] Lustre: server umount lustre-MDT0001 complete [ 6128.653726] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6129.189665] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:15:58 (1756815358) [ 6131.362574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6131.534959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6132.806884] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6135.643694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6135.800776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6137.131373] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6138.922668] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6140.182481] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6142.447978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6144.456371] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6146.173467] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6146.813987] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6157.690329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6161.376647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6161.379438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6161.384085] Lustre: Skipped 1 previous similar message [ 6161.386082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6161.387728] Lustre: Skipped 1 previous similar message [ 6165.101032] LustreError: 420328:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6165.102649] LustreError: 420328:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6165.159786] Lustre: server umount lustre-OST0000 complete [ 6166.495816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6166.499953] LustreError: Skipped 1 previous similar message [ 6166.502314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6166.507101] Lustre: Skipped 1 previous similar message [ 6166.508829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6166.510227] Lustre: Skipped 2 previous similar messages [ 6172.411390] LustreError: 420530:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6172.413787] LustreError: 420530:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6172.443248] Lustre: server umount lustre-MDT0000 complete [ 6172.641201] LustreError: 417870:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6172.644872] LustreError: 417870:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 6173.689841] LustreError: 420329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815403 with bad export cookie 14716897217338951634 [ 6173.695608] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6173.853902] Lustre: server umount lustre-MDT0001 complete [ 6177.487274] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6178.482703] Key type lgssc unregistered [ 6178.609380] LNet: 421262:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6178.612446] LNetError: 421262:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6178.620276] LNet: Removed LNI 192.168.202.121@tcp [ 6178.898890] Key type .llcrypt unregistered [ 6178.900335] Key type ._llcrypt unregistered [ 6185.953558] Key type ._llcrypt registered [ 6185.954635] Key type .llcrypt registered [ 6185.993156] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6186.103339] Key type .llcrypt unregistered [ 6186.104326] Key type ._llcrypt unregistered [ 6192.659696] Key type ._llcrypt registered [ 6192.660628] Key type .llcrypt registered [ 6192.695908] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6193.121648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6193.128173] alg: No test for adler32 (adler32-zlib) [ 6193.977875] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 6194.060236] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6195.639127] Key type lgssc registered [ 6195.953604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6198.314575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6198.321027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6199.481822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6200.708814] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6203.379091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6203.493593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6205.262761] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6206.883299] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6208.993077] LustreError: 423409:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6208.996529] LustreError: 423409:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 6209.053567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6209.172869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6210.223340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6210.528534] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6214.624778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6214.627350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.631361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6215.652609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6215.652986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.658095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6219.501337] LustreError: 425464:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6219.567797] Lustre: server umount lustre-OST0000 complete [ 6222.817643] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6222.820148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6222.823872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6222.826406] Lustre: Skipped 1 previous similar message [ 6225.888390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6225.888596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.019890] LustreError: 425666:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6227.023388] LustreError: 425666:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6227.060791] Lustre: server umount lustre-MDT0000 complete [ 6228.776928] LustreError: 425378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815458 with bad export cookie 1066559267015745167 [ 6228.783933] LustreError: 425378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6228.785035] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6228.872720] LustreError: 425867:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6228.875479] LustreError: 425867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6228.968962] Lustre: server umount lustre-MDT0001 complete [ 6232.816717] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6233.760434] Key type lgssc unregistered [ 6233.884378] LNet: 426399:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6233.886430] LNetError: 426399:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6233.895269] LNet: Removed LNI 192.168.202.121@tcp [ 6234.197681] Key type .llcrypt unregistered [ 6234.198643] Key type ._llcrypt unregistered [ 6243.533062] Key type ._llcrypt registered [ 6243.533994] Key type .llcrypt registered [ 6243.569493] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6243.947775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6243.981120] alg: No test for adler32 (adler32-zlib) [ 6244.843099] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 6244.936662] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6246.527121] Key type lgssc registered [ 6246.896456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6249.595519] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6249.601257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6266.079321] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6272.223171] LustreError: 427747:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 6302.943217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6304.977380] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6306.808849] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6309.034456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6309.228618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6309.253055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6310.554981] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6313.346785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6313.475600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6314.767175] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6316.529217] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6317.727192] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6321.634856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6321.641385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6321.644360] Lustre: Skipped 1 previous similar message [ 6326.125123] LustreError: 430028:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6326.195728] Lustre: server umount lustre-OST0000 complete [ 6328.800156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6328.802213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.805197] Lustre: Skipped 1 previous similar message [ 6328.806855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6331.872971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6331.873695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6333.627908] LustreError: 430229:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6333.629828] LustreError: 430229:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6333.659799] Lustre: server umount lustre-MDT0000 complete [ 6334.960535] LustreError: 427755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815564 with bad export cookie 8468638970641328325 [ 6334.965033] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6335.041380] LustreError: 430433:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6335.044471] LustreError: 430433:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6335.151760] Lustre: server umount lustre-MDT0001 complete [ 6338.888551] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6340.053145] Key type lgssc unregistered [ 6340.201500] LNet: 430964:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6340.205336] LNetError: 430964:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6340.212373] LNet: Removed LNI 192.168.202.121@tcp [ 6340.570104] Key type .llcrypt unregistered [ 6340.570996] Key type ._llcrypt unregistered [ 6349.530899] Key type ._llcrypt registered [ 6349.531716] Key type .llcrypt registered [ 6349.563345] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6349.978676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6349.989199] alg: No test for adler32 (adler32-zlib) [ 6350.836969] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 6350.915444] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6352.495105] Key type lgssc registered [ 6352.816546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6355.199607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6355.206927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.364875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.587117] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6360.271847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6360.384874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6361.622346] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6363.294976] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6364.486602] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6366.645684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6366.750251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6368.404433] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6370.068353] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6381.098521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6390.592931] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 6391.267688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.270321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6391.272697] Lustre: Skipped 2 previous similar messages [ 6391.274517] Lustre: Skipped 1 previous similar message [ 6391.277921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6392.450995] LustreError: 434604:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6392.479785] Lustre: server umount lustre-MDT0000 complete [ 6393.257506] LustreError: 432318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.21@tcp arrived at 1756815622 with bad export cookie 9700252925460916402 [ 6393.261309] LustreError: 432318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6393.894393] LustreError: 432318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815623 with bad export cookie 9700252925460915779 [ 6393.899112] LustreError: 432318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6393.899635] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6395.360035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6395.362077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.365642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6395.367231] Lustre: Skipped 1 previous similar message [ 6396.383983] LustreError: 432337:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.388058] LustreError: 432337:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 6398.432058] LustreError: 432337:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6398.432407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6398.436455] LustreError: 432337:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6398.442936] Lustre: Skipped 1 previous similar message [ 6399.981120] LustreError: 434808:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6399.983146] LustreError: 434808:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6400.058572] Lustre: server umount lustre-OST0000 complete [ 6402.070459] LustreError: 435164:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6402.072019] LustreError: 435164:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6402.147703] Lustre: server umount lustre-MDT0001 complete [ 6405.416517] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6406.351950] Key type lgssc unregistered [ 6406.467349] LNet: 435695:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6406.469235] LNetError: 435695:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6406.480312] LNet: Removed LNI 192.168.202.121@tcp [ 6406.751200] Key type .llcrypt unregistered [ 6406.752045] Key type ._llcrypt unregistered [ 6415.642540] Key type ._llcrypt registered [ 6415.643500] Key type .llcrypt registered [ 6415.680597] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6416.083234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6416.127331] alg: No test for adler32 (adler32-zlib) [ 6416.978977] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 6417.066458] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6418.647137] Key type lgssc registered [ 6419.026978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6421.421110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6421.427443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6422.574245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6423.743110] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6426.268519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6426.380869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6427.568111] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6429.342523] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6430.534682] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6432.653241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6432.737249] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6432.768573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6434.561499] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6435.883743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6436.173026] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6440.931914] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.935698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6440.937015] Lustre: Skipped 1 previous similar message [ 6442.817316] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) [ 6444.089709] LustreError: 439317:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6444.110875] Lustre: server umount lustre-MDT0001 complete [ 6446.048198] LustreError: 437070:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.048613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6446.049689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.049696] Lustre: Skipped 1 previous similar message [ 6446.050342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6446.052364] LustreError: 437070:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 6451.168225] LustreError: 437070:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.168726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6451.821128] LustreError: 439522:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6451.822736] LustreError: 439522:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6451.882977] Lustre: server umount lustre-OST0000 complete [ 6453.133080] LustreError: 439723:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6453.134664] LustreError: 439723:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6453.224693] Lustre: server umount lustre-MDT0000 complete [ 6457.233690] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6458.169070] Key type lgssc unregistered [ 6458.287402] LNet: 440405:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6458.289283] LNetError: 440405:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6458.297294] LNet: Removed LNI 192.168.202.121@tcp [ 6458.579805] Key type .llcrypt unregistered [ 6458.580676] Key type ._llcrypt unregistered [ 6466.019642] Key type ._llcrypt registered [ 6466.020573] Key type .llcrypt registered [ 6466.059386] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 6471.685138] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6472.159640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6472.166818] alg: No test for adler32 (adler32-zlib) [ 6473.016221] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 6473.102942] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6474.679132] Key type lgssc registered [ 6475.020862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6477.588218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6479.491844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6481.305632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6483.178356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6485.816285] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6488.469136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6490.532104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6490.550740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6490.555793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.638028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6491.647730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6491.681897] Lustre: lustre-MDT0000: new disk, initializing [ 6491.703929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6491.709617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6492.924367] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6496.292142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6498.386579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6498.409710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6498.520686] Lustre: lustre-OST0000: new disk, initializing [ 6498.523463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6498.526093] Lustre: Skipped 1 previous similar message [ 6498.547494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.391851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6499.395847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6499.403335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6500.237357] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6503.237594] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6506.491201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6508.674167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6508.695959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6508.727238] Lustre: 444348:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6508.751848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6508.790279] Lustre: lustre-MDT0001: new disk, initializing [ 6508.830383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6508.842442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6508.845303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6510.017636] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6513.348705] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6513.408559] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6514.638147] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6518.798066] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 4 sec [ 6522.849448] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6522.849928] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.852149] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6522.854286] Lustre: Skipped 1 previous similar message [ 6522.856814] Lustre: Skipped 1 previous similar message [ 6526.400286] LustreError: 447796:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6526.425630] Lustre: server umount lustre-MDT0001 complete [ 6533.090274] LustreError: 444354:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6533.097762] LustreError: 444354:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 6538.209648] LustreError: 444354:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6538.213435] LustreError: 444354:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6540.255875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6540.258053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6540.261555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6540.263139] Lustre: Skipped 1 previous similar message [ 6543.328965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6543.329300] LustreError: 444358:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6544.750097] LustreError: 448070:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6544.751849] LustreError: 448070:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6544.821961] Lustre: server umount lustre-OST0000 complete [ 6552.194728] LustreError: 448272:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6552.197810] LustreError: 448272:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6552.307594] Lustre: server umount lustre-MDT0000 complete [ 6556.918351] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6557.928895] Key type lgssc unregistered [ 6558.057331] LNet: 448955:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6558.059469] LNetError: 448955:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6558.071286] LNet: Removed LNI 192.168.202.121@tcp [ 6558.337333] Key type .llcrypt unregistered [ 6558.338330] Key type ._llcrypt unregistered [ 6570.850603] Key type ._llcrypt registered [ 6570.851464] Key type .llcrypt registered [ 6570.883411] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6570.975328] Key type .llcrypt unregistered [ 6570.976184] Key type ._llcrypt unregistered [ 6577.097938] Key type ._llcrypt registered [ 6577.099751] Key type .llcrypt registered [ 6577.140696] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6577.473755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6577.508243] alg: No test for adler32 (adler32-zlib) [ 6578.352169] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 6578.430994] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6580.007174] Key type lgssc registered [ 6580.325263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6582.782342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6584.814668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6586.689261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6588.751710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6588.770292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6588.774936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6589.860642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6589.870632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6589.906406] Lustre: lustre-MDT0000: new disk, initializing [ 6589.930833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6589.937754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6591.137120] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6595.002928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6595.025525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6595.054846] Lustre: 452275:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6595.076025] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6595.077423] Lustre: Skipped 1 previous similar message [ 6595.111142] Lustre: lustre-MDT0001: new disk, initializing [ 6595.139850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6595.150509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6595.153122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6596.302045] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6599.162090] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6600.162934] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6602.249009] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6602.272775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6602.365434] Lustre: lustre-OST0000: new disk, initializing [ 6602.366953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6602.396324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6604.035070] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6606.326525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6606.329436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6606.361106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6606.862604] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6610.399952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6610.401740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.404807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.426080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.426446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.429550] Lustre: Skipped 1 previous similar message [ 6611.431967] Lustre: Skipped 2 previous similar messages [ 6615.043241] LustreError: 454713:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6615.071889] Lustre: server umount lustre-MDT0000 complete [ 6616.276741] LustreError: 452264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815845 with bad export cookie 13731189989417947977 [ 6616.279363] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6616.282266] LustreError: 452264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6616.337181] LustreError: 454917:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6616.338665] LustreError: 454917:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6616.419688] Lustre: server umount lustre-MDT0001 complete [ 6627.824604] LustreError: 455118:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6627.826228] LustreError: 455118:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6627.891851] Lustre: server umount lustre-OST0000 complete [ 6631.060539] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:24:20 (1756815860) [ 6631.405860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6633.567838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.765124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.053731] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6637.717287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6637.848129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6639.071030] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6640.758652] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6641.842865] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6643.905629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6643.931565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6644.000226] Lustre: Found index 0 for lustre-OST0000, updating log [ 6645.095277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6645.817623] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6647.575386] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6650.335838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6650.338083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.342069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6654.762230] LustreError: 458197:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6654.766417] LustreError: 458197:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6654.838169] Lustre: server umount lustre-OST0000 complete [ 6655.719031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6656.480282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6656.480807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6656.483901] Lustre: Skipped 2 previous similar messages [ 6656.486334] Lustre: Skipped 4 previous similar messages [ 6662.459473] LustreError: 458459:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6662.461522] LustreError: 458459:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6662.484900] Lustre: server umount lustre-MDT0000 complete [ 6662.624045] LustreError: 455946:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6662.628075] LustreError: 455946:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 6663.676332] LustreError: 458110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815893 with bad export cookie 13731189989417949356 [ 6663.679974] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6663.681549] LustreError: 458110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6663.816510] Lustre: server umount lustre-MDT0001 complete [ 6666.355666] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:24:55 (1756815895) [ 6666.681333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6666.989517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6667.313304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6667.655701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6667.978071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6668.297143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6668.627206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6668.962356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6669.277385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6669.613395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6669.953484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6670.299069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6670.623295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6670.919271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6671.255477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6671.593411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6671.924551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6672.268511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6672.611675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6672.928364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6673.265244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6673.616142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6673.946282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6674.259319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6674.581786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6674.895242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6675.205339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6675.524905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6675.836273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6676.169374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6676.490337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6676.791827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6677.124432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6677.455862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6677.773334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6678.076812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6678.401172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6678.743177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6679.077741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6679.383583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6679.704319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6680.010534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6680.312306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6680.604191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6680.921188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6681.245513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6681.571194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6681.885102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6682.202780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6682.516074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6682.826787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6685.188114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6685.328445] LustreError: 462039:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.350870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6685.352464] Lustre: Skipped 1 previous similar message [ 6686.572287] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6689.305810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6690.717584] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6692.462505] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6693.578172] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6695.634180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6695.659175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6695.737837] Lustre: Found index 0 for lustre-OST0000, updating log [ 6697.501976] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6699.199576] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6699.836069] LustreError: 464344:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6699.837604] LustreError: 464344:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6699.895070] Lustre: server umount lustre-OST0000 complete [ 6701.011512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6701.012363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6701.014117] Lustre: Skipped 1 previous similar message [ 6701.015513] Lustre: Skipped 2 previous similar messages [ 6712.288365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6712.291139] Lustre: Skipped 6 previous similar messages [ 6715.359094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6715.416158] Lustre: server umount lustre-MDT0000 complete [ 6716.384214] LustreError: 462038:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6716.390477] LustreError: 462038:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6716.566044] LustreError: 462019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815946 with bad export cookie 13731189989417950420 [ 6716.569080] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6716.627893] LustreError: 464752:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6716.629641] LustreError: 464752:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6716.704633] Lustre: server umount lustre-MDT0001 complete [ 6717.557579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6722.310145] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 6725.082632] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6728.173246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6730.334838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6732.179974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6734.107788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6736.264315] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 08:26:05 (1756815965) [ 6779.919156] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6780.859587] Key type lgssc unregistered [ 6780.980490] LNet: 468336:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6780.982908] LNetError: 468336:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6911.078614] LNet: Removed LNI 192.168.202.121@tcp [ 6911.388572] Key type .llcrypt unregistered [ 6911.389851] Key type ._llcrypt unregistered [ 6917.370061] Key type ._llcrypt registered [ 6917.372064] Key type .llcrypt registered [ 6917.412973] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6917.816902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6917.851432] alg: No test for adler32 (adler32-zlib) [ 6918.698610] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 6918.779893] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6920.359113] Key type lgssc registered [ 6920.726734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6922.137064] LNet: Added LNI 192.168.202.121@tcp100 [8/256/0/180] [ 6922.640956] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 6928.163682] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6929.107159] Key type lgssc unregistered [ 6929.260914] LNet: 470203:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6929.262933] LNetError: 470203:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6929.264691] LNet: Removed LNI 192.168.202.121@tcp [ 6929.269249] LNet: Removed LNI 192.168.202.121@tcp100 [ 6929.504331] Key type .llcrypt unregistered [ 6929.505387] Key type ._llcrypt unregistered [ 6936.510666] Key type ._llcrypt registered [ 6936.511524] Key type .llcrypt registered [ 6936.546286] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 6941.886101] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6942.167195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6942.193057] alg: No test for adler32 (adler32-zlib) [ 6943.028897] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 6943.100019] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6944.671113] Key type lgssc registered [ 6944.979208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6947.470323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6949.330847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6951.056101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6952.884283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6954.832186] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 08:29:44 (1756816184) [ 6960.262359] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6961.307739] Key type lgssc unregistered [ 6961.455365] LNet: 474213:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6961.457312] LNetError: 474213:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6961.465377] LNet: Removed LNI 192.168.202.121@tcp [ 6961.758354] Key type .llcrypt unregistered [ 6961.759279] Key type ._llcrypt unregistered [ 6967.775346] Key type ._llcrypt registered [ 6967.776300] Key type .llcrypt registered [ 6967.811549] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6968.156799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6968.169293] alg: No test for adler32 (adler32-zlib) [ 6969.027536] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 6969.116429] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6970.695138] Key type lgssc registered [ 6971.050681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6973.942683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6973.963487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6973.968691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6975.072054] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6975.083128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6975.121439] Lustre: lustre-MDT0000: new disk, initializing [ 6975.148587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6975.154343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6976.401153] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6980.397060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6980.420193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6980.449461] Lustre: 475384:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6980.476308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6980.477810] Lustre: Skipped 1 previous similar message [ 6980.515774] Lustre: lustre-MDT0001: new disk, initializing [ 6980.552162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6980.563861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6980.566917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6981.768483] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6984.645507] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6985.716084] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6987.791479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6987.821029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6987.917566] Lustre: lustre-OST0000: new disk, initializing [ 6987.919411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6987.948416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6988.450302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6988.453058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6988.490864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6989.701972] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6992.545569] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6993.475935] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing lnet_if_list [ 6994.834390] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6994.842716] LNet: Added LNI 192.168.202.121@tcp42 [8/256/0/180] [ 7004.129068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7004.130205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7004.130956] LustreError: Skipped 1 previous similar message [ 7004.134106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7004.135818] Lustre: Skipped 1 previous similar message [ 7006.634078] LustreError: 478759:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7006.688329] Lustre: server umount lustre-OST0000 complete [ 7009.248806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7009.249831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7009.252704] Lustre: Skipped 2 previous similar messages [ 7009.257120] Lustre: Skipped 1 previous similar message [ 7013.956976] LustreError: 478964:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7013.959450] LustreError: 478964:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7013.991145] Lustre: server umount lustre-MDT0000 complete [ 7015.253067] LustreError: 475374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756816244 with bad export cookie 12663183230957363432 [ 7015.255601] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7015.255927] LustreError: 475374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7015.314505] LustreError: 479165:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7015.315953] LustreError: 479165:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7015.393798] Lustre: server umount lustre-MDT0001 complete [ 7018.817770] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7020.007434] Key type lgssc unregistered [ 7020.155143] LNet: 479704:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7020.157146] LNetError: 479704:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7020.158920] LNet: Removed LNI 192.168.202.121@tcp [ 7020.168284] LNet: Removed LNI 192.168.202.121@tcp42 [ 7020.420617] Key type .llcrypt unregistered [ 7020.421478] Key type ._llcrypt unregistered [ 7027.909792] Key type ._llcrypt registered [ 7027.910817] Key type .llcrypt registered [ 7027.954244] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7028.393253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7028.408850] alg: No test for adler32 (adler32-zlib) [ 7029.255321] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 7029.345992] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7030.927108] Key type lgssc registered [ 7031.239455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7033.520242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7033.528387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.675232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7035.854363] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7038.532712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7038.649425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7039.841319] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7041.397744] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7042.444290] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7044.422316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7044.520565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7046.106552] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7047.652822] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7051.922721] loop0: detected capacity change from 0 to 204800000 [ 7051.978537] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7051.981719] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7051.984917] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7051.988059] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7052.007442] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7052.025699] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7052.038611] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7052.059804] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7052.083350] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7052.117660] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7053.898528] loop0: detected capacity change from 0 to 204800000 [ 7053.948508] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7053.996074] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7055.533591] loop0: detected capacity change from 0 to 204800000 [ 7057.306642] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7057.331198] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7057.371552] Lustre: fs15246-OST0000: new disk, initializing [ 7057.373407] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7057.390309] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7059.069624] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7062.732858] loop1: detected capacity change from 0 to 204800000 [ 7064.503580] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7064.524289] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7064.551255] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7064.570101] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7064.608814] Lustre: fs15246-MDT0000: new disk, initializing [ 7064.648320] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7064.656223] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7064.686044] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7065.695876] Lustre: 485300:0:(client.c:1628:after_reply()) @@@ resending request on EINPROGRESS req@000000000eaea1cc x1842155364576000/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1756816311 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7065.792968] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7074.783577] LustreError: 485300:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7074.786548] LustreError: 485300:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7074.789118] LustreError: 485300:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7076.327683] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.331724] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7078.714110] LustreError: 486422:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7078.740883] Lustre: server umount fs15246-MDT0000 complete [ 7081.450518] LustreError: 480921:0:(ldlm_lib.c:1132:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7081.454083] LustreError: 480921:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 7081.587493] LustreError: 486923:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7081.589103] LustreError: 486923:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7081.602554] Lustre: server umount fs15246-OST0000 complete [ 7086.564040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.565624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7086.565736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7086.566966] Lustre: Skipped 1 previous similar message [ 7091.681358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7091.683066] Lustre: Skipped 2 previous similar messages [ 7096.803419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7096.807310] Lustre: Skipped 1 previous similar message [ 7099.359108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7099.373958] LustreError: 487425:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7099.375463] LustreError: 487425:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7099.441693] Lustre: server umount lustre-OST0000 complete [ 7101.921621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.925371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.926734] Lustre: Skipped 1 previous similar message [ 7106.750053] LustreError: 487629:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7106.752505] LustreError: 487629:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7106.780825] Lustre: server umount lustre-MDT0000 complete [ 7107.040441] LustreError: 480925:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7107.044942] LustreError: 480925:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 7107.962623] LustreError: 480905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756816337 with bad export cookie 5930667232680902045 [ 7107.966662] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7107.967825] LustreError: 480905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7108.136531] Lustre: server umount lustre-MDT0001 complete [ 7111.447866] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7112.537297] Key type lgssc unregistered [ 7112.675422] LNet: 488361:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7112.677699] LNetError: 488361:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7112.684466] LNet: Removed LNI 192.168.202.121@tcp [ 7113.006826] Key type .llcrypt unregistered [ 7113.007870] Key type ._llcrypt unregistered [ 7119.902269] Key type ._llcrypt registered [ 7119.903203] Key type .llcrypt registered [ 7119.934437] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7120.264720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7120.312906] alg: No test for adler32 (adler32-zlib) [ 7121.162430] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 7121.242954] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7122.823144] Key type lgssc registered [ 7123.162841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7125.390951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7126.840058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7128.429469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7129.881755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7134.670747] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 7137.632667] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7141.034216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7143.110959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7144.994820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7147.029056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7149.220955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7149.240544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7149.245939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.339907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7150.351335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7150.391439] Lustre: lustre-MDT0000: new disk, initializing [ 7150.417081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7150.424477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7151.691580] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7155.675383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7155.699624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7155.727064] Lustre: 492760:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7155.750893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7155.752288] Lustre: Skipped 1 previous similar message [ 7155.786726] Lustre: lustre-MDT0001: new disk, initializing [ 7155.815696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7155.826836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7155.829550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7156.949233] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7159.827668] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7160.960194] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7162.993776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7163.015378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7163.103247] Lustre: lustre-OST0000: new disk, initializing [ 7163.106198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7163.134665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7164.770420] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7167.680405] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7168.966364] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7171.581928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7171.584632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7171.623557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7172.102026] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7173.285485] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7173.354880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7176.673909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.674205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.675707] LustreError: Skipped 1 previous similar message [ 7176.676288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.679312] Lustre: Skipped 1 previous similar message [ 7180.013220] LustreError: 495718:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7180.069554] Lustre: server umount lustre-OST0000 complete [ 7181.794037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7181.794203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.794917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7181.794922] Lustre: Skipped 1 previous similar message [ 7181.801298] Lustre: Skipped 1 previous similar message [ 7186.912844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7186.914542] Lustre: Skipped 2 previous similar messages [ 7187.322824] LustreError: 495919:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7187.324389] LustreError: 495919:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7187.357162] Lustre: server umount lustre-MDT0000 complete [ 7188.433993] LustreError: 493541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756816418 with bad export cookie 9902260932904195318 [ 7188.437197] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7188.438649] LustreError: 493541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7188.502949] LustreError: 496121:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7188.504527] LustreError: 496121:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7188.579825] Lustre: server umount lustre-MDT0001 complete [ 7195.795378] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 7198.790992] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7202.102764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7204.576514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7206.442071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7208.493138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7209.017986] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7023 sec ========== 08:33:58 (1756816438) [ 7209.571302] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:33:58 (1756816438) === [ 7210.538681] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:33:59 (1756816439) === [ 7221.777545] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7222.728296] Key type lgssc unregistered [ 7222.851331] LNet: 500969:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7222.853156] LNetError: 500969:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7222.863308] LNet: Removed LNI 192.168.202.121@tcp [ 7223.130176] Key type .llcrypt unregistered [ 7223.131144] Key type ._llcrypt unregistered