[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 264776301 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.415668] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.417898] pid_max: default: 32768 minimum: 301 [ 0.419416] Security Framework initialized [ 0.420503] SELinux: Initializing. [ 0.421287] SELinux: Starting in permissive mode [ 0.422495] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.425747] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.427570] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.428854] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.430637] Initializing cgroup subsys memory [ 0.431501] Initializing cgroup subsys devices [ 0.432402] Initializing cgroup subsys freezer [ 0.433220] Initializing cgroup subsys net_cls [ 0.434315] Initializing cgroup subsys blkio [ 0.435519] Initializing cgroup subsys perf_event [ 0.436882] Initializing cgroup subsys hugetlb [ 0.437707] Initializing cgroup subsys pids [ 0.438529] Initializing cgroup subsys net_prio [ 0.439559] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.441822] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.443401] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.445063] tlb_flushall_shift: 6 [ 0.446256] FEATURE SPEC_CTRL Present [ 0.447354] FEATURE IBPB_SUPPORT Present [ 0.448495] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.450562] Spectre V2 : Vulnerable [ 0.451697] Speculative Store Bypass: Vulnerable [ 0.454005] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.461616] ACPI: Core revision 20130517 [ 0.464747] ACPI: All ACPI Tables successfully acquired [ 0.466435] ftrace: allocating 30294 entries in 119 pages [ 0.522270] Enabling x2apic [ 0.523146] Enabled x2apic [ 0.524357] Switched APIC routing to physical x2apic. [ 0.527594] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.529585] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.532740] TSC deadline timer enabled [ 0.532928] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.535497] ... version: 2 [ 0.536625] ... bit width: 48 [ 0.537871] ... generic registers: 4 [ 0.539200] ... value mask: 0000ffffffffffff [ 0.540700] ... max period: 00007fffffffffff [ 0.542178] ... fixed-purpose events: 3 [ 0.543368] ... event mask: 000000070000000f [ 0.545013] KVM setup paravirtual spinlock [ 0.550179] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.553548] KVM setup async PF for cpu 1 [ 0.554910] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.557238] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.560169] KVM setup async PF for cpu 2 [ 0.548279] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.560587] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.563714] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.565575] Brought up 4 CPUs [ 0.565603] KVM setup async PF for cpu 3 [ 0.565607] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.567662] smpboot: Max logical packages: 1 [ 0.568473] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.571304] devtmpfs: initialized [ 0.572173] x86/mm: Memory block size: 128MB [ 0.575379] EVM: security.selinux [ 0.575988] EVM: security.ima [ 0.576555] EVM: security.capability [ 0.578576] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.580030] NET: Registered protocol family 16 [ 0.581030] cpuidle: using governor haltpoll [ 0.582583] ACPI: bus type PCI registered [ 0.583525] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.584850] PCI: Using configuration type 1 for base access [ 0.586690] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.593532] ACPI: Added _OSI(Module Device) [ 0.594843] ACPI: Added _OSI(Processor Device) [ 0.596260] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.597219] ACPI: Added _OSI(Processor Aggregator Device) [ 0.598404] ACPI: Added _OSI(Linux-Dell-Video) [ 0.600033] ACPI: EC: Look up EC in DSDT [ 0.602099] ACPI: Interpreter enabled [ 0.603070] ACPI: (supports S0 S3 S4 S5) [ 0.603854] ACPI: Using IOAPIC for interrupt routing [ 0.604868] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.606839] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.611401] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.612638] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.613931] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.615185] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.617746] acpiphp: Slot [2] registered [ 0.618531] acpiphp: Slot [3] registered [ 0.619297] acpiphp: Slot [4] registered [ 0.620077] acpiphp: Slot [5] registered [ 0.620877] acpiphp: Slot [6] registered [ 0.621665] acpiphp: Slot [7] registered [ 0.622431] acpiphp: Slot [8] registered [ 0.623184] acpiphp: Slot [9] registered [ 0.624019] acpiphp: Slot [10] registered [ 0.624774] acpiphp: Slot [11] registered [ 0.625516] acpiphp: Slot [12] registered [ 0.626255] acpiphp: Slot [13] registered [ 0.627122] acpiphp: Slot [14] registered [ 0.627916] acpiphp: Slot [15] registered [ 0.628641] acpiphp: Slot [16] registered [ 0.629392] acpiphp: Slot [17] registered [ 0.630238] acpiphp: Slot [18] registered [ 0.630964] acpiphp: Slot [19] registered [ 0.631699] acpiphp: Slot [20] registered [ 0.632448] acpiphp: Slot [21] registered [ 0.633288] acpiphp: Slot [22] registered [ 0.634080] acpiphp: Slot [23] registered [ 0.635418] acpiphp: Slot [24] registered [ 0.636397] acpiphp: Slot [25] registered [ 0.637164] acpiphp: Slot [26] registered [ 0.637938] acpiphp: Slot [27] registered [ 0.638740] acpiphp: Slot [28] registered [ 0.639582] acpiphp: Slot [29] registered [ 0.640380] acpiphp: Slot [30] registered [ 0.641196] acpiphp: Slot [31] registered [ 0.641958] PCI host bridge to bus 0000:00 [ 0.642794] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.644012] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.645310] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.646594] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.648105] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.649477] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.650534] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.651058] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.651636] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.655550] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.657161] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.658656] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.659875] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.661190] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.663670] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.664034] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.665454] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.667027] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.668355] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.675473] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.676640] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.677734] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.681132] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.684074] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.693793] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.699600] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.702330] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.704897] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.715206] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.718636] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.723743] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.726079] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.736266] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.740679] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.743326] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.746260] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.757693] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.763109] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.766319] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.769455] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.782136] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.786862] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.788955] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.791991] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.806144] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.811777] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.813897] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.816028] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.818738] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.820697] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.823990] vgaarb: loaded [ 0.825171] SCSI subsystem initialized [ 0.826346] ACPI: bus type USB registered [ 0.827611] usbcore: registered new interface driver usbfs [ 0.829378] usbcore: registered new interface driver hub [ 0.831082] usbcore: registered new device driver usb [ 0.832870] PCI: Using ACPI for IRQ routing [ 0.834156] PCI: pci_cache_line_size set to 64 bytes [ 0.834424] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.834429] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.834431] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 0.834771] NetLabel: Initializing [ 0.835980] NetLabel: domain hash size = 128 [ 0.837277] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.838815] NetLabel: unlabeled traffic allowed by default [ 0.840828] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.842479] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.847608] amd_nb: Cannot enumerate AMD northbridges [ 0.849353] Switched to clocksource kvm-clock [ 0.869046] pnp: PnP ACPI init [ 0.870774] ACPI: bus type PNP registered [ 0.872099] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.872184] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.872231] pnp 00:02: [dma 2] [ 0.872264] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.872366] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.872442] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.872517] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.873010] pnp: PnP ACPI: found 6 devices [ 0.874353] ACPI: bus type PNP unregistered [ 0.884789] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.884796] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.884799] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.884801] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.884804] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 0.884873] NET: Registered protocol family 2 [ 0.886823] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.889634] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 0.892628] TCP: Hash tables configured (established 32768 bind 32768) [ 0.894872] TCP: reno registered [ 0.895665] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 0.896900] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 0.898708] NET: Registered protocol family 1 [ 0.900035] RPC: Registered named UNIX socket transport module. [ 0.901940] RPC: Registered udp transport module. [ 0.903501] RPC: Registered tcp transport module. [ 0.904985] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.907038] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.908857] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.910565] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.912663] PCI: CLS 0 bytes, default 64 [ 0.912829] Unpacking initramfs... [ 2.311773] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.315633] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.317719] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.320397] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.322633] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.324416] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.326362] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.330681] cryptomgr_test (50) used greatest stack depth: 14128 bytes left [ 2.331559] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.331620] Initialise system trusted keyring [ 2.370865] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.373287] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.378025] zpool: loaded [ 2.378546] zbud: loaded [ 2.379310] VFS: Disk quotas dquot_6.6.0 [ 2.380721] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.382699] NFS: Registering the id_resolver key type [ 2.383689] Key type id_resolver registered [ 2.385077] Key type id_legacy registered [ 2.385856] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.388472] Key type big_key registered [ 2.389697] SELinux: Registering netfilter hooks [ 2.391620] cryptomgr_test (56) used greatest stack depth: 13968 bytes left [ 2.396379] cryptomgr_test (59) used greatest stack depth: 13664 bytes left [ 2.397013] NET: Registered protocol family 38 [ 2.397023] Key type asymmetric registered [ 2.397026] Asymmetric key parser 'x509' registered [ 2.397141] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.397254] io scheduler noop registered [ 2.397258] io scheduler deadline registered (default) [ 2.397328] io scheduler cfq registered [ 2.397334] io scheduler mq-deadline registered [ 2.397338] io scheduler kyber registered [ 2.399435] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.399445] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.411233] intel_idle: does not run on family 6 model 62 [ 2.411389] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.413921] ACPI: Power Button [PWRF] [ 2.415549] GHES: HEST is not enabled! [ 2.474545] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.537481] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.681546] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.780831] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.916827] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.946184] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.975282] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.978851] Non-volatile memory driver v1.3 [ 2.980109] Linux agpgart interface v0.103 [ 2.982016] crash memory driver: version 1.1 [ 2.983594] nbd: registered device at major 43 [ 2.987936] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 2.987966] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 2.988000] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 2.988031] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 2.988076] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 2.994855] virtio_blk virtio1: [vda] 60184 512-byte logical blocks (30.8 MB/29.3 MiB) [ 2.999621] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 2.999652] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 2.999685] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 2.999711] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 2.999754] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.005718] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.012936] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.012987] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.013020] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.013050] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.013080] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.019705] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.029133] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.029167] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.029191] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.029214] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.029237] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.035658] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.040751] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.040796] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.040836] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.040875] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.040917] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.046654] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.052103] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.052131] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.052153] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.052175] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.052203] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.057995] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.063005] rdac: device handler registered [ 3.064594] hp_sw: device handler registered [ 3.066238] emc: device handler registered [ 3.067443] libphy: Fixed MDIO Bus: probed [ 3.072625] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.075206] ehci-pci: EHCI PCI platform driver [ 3.077062] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.078776] ohci-pci: OHCI PCI platform driver [ 3.080386] uhci_hcd: USB Universal Host Controller Interface driver [ 3.082834] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.087008] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.088346] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.089844] mousedev: PS/2 mouse device common for all mice [ 3.092294] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.097547] rtc_cmos 00:05: RTC can wake from S4 [ 3.100021] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.102486] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.107117] hidraw: raw HID events driver (C) Jiri Kosina [ 3.109130] usbcore: registered new interface driver usbhid [ 3.111079] usbhid: USB HID core driver [ 3.112489] drop_monitor: Initializing network drop monitor service [ 3.114511] Netfilter messages via NETLINK v0.30. [ 3.115863] TCP: cubic registered [ 3.116775] Initializing XFRM netlink socket [ 3.118108] NET: Registered protocol family 10 [ 3.119826] NET: Registered protocol family 17 [ 3.120676] Key type dns_resolver registered [ 3.122385] mce: Using 10 MCE banks [ 3.123910] Loading compiled-in X.509 certificates [ 3.126704] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.129892] registered taskstats version 1 [ 3.133591] modprobe (70) used greatest stack depth: 13456 bytes left [ 3.137593] Key type trusted registered [ 3.143037] Key type encrypted registered [ 3.144175] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.147466] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.150235] rtc_cmos 00:05: setting system clock to 2024-06-20 02:47:18 UTC (1718851638) [ 3.153905] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.156690] Write protecting the kernel read-only data: 12288k [ 3.158843] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.161306] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.170570] random: systemd: uninitialized urandom read (16 bytes read) [ 3.172804] random: systemd: uninitialized urandom read (16 bytes read) [ 3.174051] random: systemd: uninitialized urandom read (16 bytes read) [ 3.177190] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.182464] systemd[1]: Detected virtualization kvm. [ 3.184010] systemd[1]: Detected architecture x86-64. [ 3.184972] systemd[1]: Running in initial RAM disk. [ 3.187886] systemd[1]: No hostname configured. [ 3.188805] systemd[1]: Set hostname to . [ 3.189800] random: systemd: uninitialized urandom read (16 bytes read) [ 3.192021] systemd[1]: Initializing machine ID from random generator. [ 3.205994] ln (87) used greatest stack depth: 13008 bytes left [ 3.237911] random: systemd: uninitialized urandom read (16 bytes read) [ 3.240528] random: systemd: uninitialized urandom read (16 bytes read) [ 3.242319] random: systemd: uninitialized urandom read (16 bytes read) [ 3.243690] random: systemd: uninitialized urandom read (16 bytes read) [ 3.245998] random: systemd: uninitialized urandom read (16 bytes read) [ 3.248057] random: systemd: uninitialized urandom read (16 bytes read) [ 3.257913] systemd[1]: Created slice Root Slice. [ 3.260794] systemd[1]: Listening on Journal Socket. [ 3.263961] systemd[1]: Created slice System Slice. [ 3.269115] systemd[1]: Starting Journal Service... [ 3.273491] systemd[1]: Starting Load Kernel Modules... [ 3.277056] systemd[1]: Reached target Slices. [ 3.280115] systemd[1]: Listening on udev Control Socket. [ 3.283346] systemd[1]: Reached target Local File Systems. [ 3.288683] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.296626] systemd[1]: Starting Setup Virtual Console... [ 3.301286] systemd[1]: Reached target Swap. [ 3.306787] systemd[1]: Starting dracut cmdline hook... [ 3.311112] systemd[1]: Listening on udev Kernel Socket. [ 3.315895] systemd[1]: Reached target Sockets. [ 3.320435] systemd[1]: Reached target Timers. [ 3.325937] systemd[1]: Started Journal Service. [ 3.330452] tsc: Refined TSC clocksource calibration: 2399.981 MHz [ 3.474026] random: fast init done [ 3.682692] libata version 3.00 loaded. [ 3.684878] ata_piix 0000:00:01.1: version 2.13 [ 3.694949] scsi host0: ata_piix [ 3.700396] scsi host1: ata_piix [ 3.701203] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.702366] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 3.773661] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 3.774987] ip (343) used greatest stack depth: 12464 bytes left [ 3.934121] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.073431] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6.786126] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 7.229911] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 7.504658] SELinux: Disabled at runtime. [ 7.506394] SELinux: Unregistering netfilter hooks [ 7.593898] ip_tables: (C) 2000-2006 Netfilter Core Team [ 7.599719] systemd[1]: Inserted module 'ip_tables' [ 7.717256] kdump-dep-gener (548) used greatest stack depth: 12368 bytes left [ 8.180625] systemd-journald[563]: Received request to flush runtime journal from PID 1 [ 8.351435] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 8.369288] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.385919] cryptd: max_cpu_qlen set to 1000 [ 8.426364] AVX version of gcm_enc/dec engaged. [ 8.428451] AES CTR mode by8 optimization enabled [ 8.459020] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.480794] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 8.486779] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 8.493068] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 8.608895] EDAC MC: Ver: 3.0.0 [ 8.620370] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 8.620381] EDAC sbridge: Ver: 1.1.2 [ 10.998621] mount.nfs (770) used greatest stack depth: 10704 bytes left [ 21.389065] device-mapper: uevent: version 1.0.3 [ 21.390963] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 25.495744] libcfs: loading out-of-tree module taints kernel. [ 25.497203] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 25.523858] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 30.080103] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 30.263829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 30.268584] alg: No test for adler32 (adler32-zlib) [ 31.138253] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 31.288860] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 31.290198] LNet: Accept secure, port 988 [ 32.831493] Key type lgssc registered [ 33.093617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 35.830554] icp: module license 'CDDL' taints kernel. [ 35.832868] Disabling lock debugging due to kernel taint [ 38.460825] ZFS: Loaded module v0.8.6-1, ZFS pool version 5000, ZFS filesystem version 5 [ 41.333940] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 45.562486] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 47.427121] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 47.429724] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 49.516535] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 49.519533] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 52.446752] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 55.404942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 55.423711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 55.430623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 56.501229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 56.508188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 56.540524] Lustre: lustre-MDT0000: new disk, initializing [ 56.558511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 56.565113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 56.591601] mount.lustre (7100) used greatest stack depth: 10000 bytes left [ 57.280094] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 61.341825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 61.368915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 61.411005] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 61.419227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 61.421383] Lustre: Skipped 1 previous similar message [ 61.460996] Lustre: lustre-MDT0001: new disk, initializing [ 61.476805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 61.488211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 61.493162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 62.236707] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 64.427730] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 67.833553] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 67.840461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 67.869665] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 67.874618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 67.963145] Lustre: lustre-OST0000: new disk, initializing [ 67.967026] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 67.987932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 69.675989] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 70.181949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 70.188106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 70.205175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 74.420546] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 74.424903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.444656] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 74.447570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 74.468989] Lustre: lustre-OST0001: new disk, initializing [ 74.470759] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 74.482094] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 75.705059] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 77.723635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 77.726694] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 77.735883] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 80.462255] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 83.275614] random: crng init done [ 86.745203] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 93.215596] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing check_logdir /tmp/testlogs/ [ 94.153926] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing yml_node [ 95.216660] Lustre: DEBUG MARKER: Client: 2.15.63.196 [ 95.893872] Lustre: DEBUG MARKER: MDS: 2.15.63.196 [ 96.554438] Lustre: DEBUG MARKER: OSS: 2.15.63.196 [ 96.956328] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jun 19 22:48:52 EDT 2024 [ 100.937637] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 101.363457] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 112.792060] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 112.792338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 112.792340] Lustre: Skipped 1 previous similar message [ 112.805190] Lustre: Skipped 3 previous similar messages [ 117.799855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 117.802166] Lustre: Skipped 5 previous similar messages [ 118.228448] Lustre: server umount lustre-MDT0000 complete [ 119.295221] LustreError: 8264:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718851754 with bad export cookie 2704434893257614807 [ 119.296279] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 119.304550] LustreError: 8264:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 122.807726] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 122.807768] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 122.807897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 122.822300] LustreError: Skipped 1 previous similar message [ 123.809914] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 123.814001] LustreError: Skipped 1 previous similar message [ 125.409184] Lustre: server umount lustre-MDT0001 complete [ 132.496208] Lustre: server umount lustre-OST0000 complete [ 139.611879] Lustre: server umount lustre-OST0001 complete [ 140.953790] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 142.445382] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 144.655020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.282950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 147.758797] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 147.762809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 149.245169] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 149.249368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 151.011267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 151.030822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.124378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 151.136506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 151.171476] Lustre: lustre-MDT0000: new disk, initializing [ 151.190106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 151.196568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 152.003399] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 155.400666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 155.420718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 155.528314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 155.549557] mount.lustre (18214) used greatest stack depth: 9616 bytes left [ 156.288177] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 158.577010] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 159.137850] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 160.860436] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 160.863680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 160.884193] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 160.887280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 160.967226] Lustre: lustre-OST0000: new disk, initializing [ 160.969680] Lustre: Skipped 1 previous similar message [ 160.972517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 160.975626] Lustre: Skipped 2 previous similar messages [ 160.996123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.998190] Lustre: Skipped 1 previous similar message [ 161.014968] mount.lustre (19274) used greatest stack depth: 9552 bytes left [ 162.220740] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 162.638767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 162.643026] Lustre: Skipped 1 previous similar message [ 162.645268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 162.657191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 164.497153] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 165.275446] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 165.323122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 166.110447] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 166.162855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 167.656304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 167.656639] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 167.661789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 167.661790] Lustre: Skipped 3 previous similar messages [ 167.669066] Lustre: Skipped 2 previous similar messages [ 172.665372] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 172.670345] LustreError: Skipped 1 previous similar message [ 172.730153] Lustre: server umount lustre-OST0000 complete [ 175.543971] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 175.547153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 175.552855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 175.555974] Lustre: Skipped 1 previous similar message [ 179.773275] Lustre: server umount lustre-MDT0000 complete [ 180.781233] LustreError: 17276:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718851816 with bad export cookie 2704434893257616284 [ 180.782748] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 180.787553] LustreError: 17276:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 180.900619] Lustre: server umount lustre-MDT0001 complete [ 182.066452] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 22:50:17 (1718851817) [ 183.778731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.923300] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 183.941458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.717407] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 186.867904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 187.739074] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 188.750412] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 189.336247] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 191.066630] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 191.070565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 192.397464] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 193.476854] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 200.078824] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 200.082009] Lustre: Skipped 1 previous similar message [ 204.888806] Lustre: server umount lustre-MDT0000 complete [ 205.839091] LustreError: 24302:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718851841 with bad export cookie 2704434893257617138 [ 205.841070] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 205.844637] LustreError: 24302:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 205.959072] Lustre: server umount lustre-MDT0001 complete [ 227.053451] Lustre: server umount lustre-OST0000 complete [ 231.191799] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 233.462683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.589873] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 233.600307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 233.604145] Lustre: Skipped 2 previous similar messages [ 234.449294] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 236.631745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 237.529134] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 238.242214] Lustre: Setting parameter general.debug_raw_pointers in log params [ 239.931179] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 239.935262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 241.226105] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 243.373163] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 243.376642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 243.395789] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 243.399221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 243.433202] Lustre: lustre-OST0001: new disk, initializing [ 243.435471] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.605419] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 244.826210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.829034] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 244.844397] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.372554] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.890000] Lustre: server umount lustre-MDT0000 complete [ 261.751873] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 261.754394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.758187] Lustre: Skipped 1 previous similar message [ 261.759750] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 261.832210] LustreError: 26552:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718851897 with bad export cookie 2704434893257617985 [ 261.833878] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 261.841043] LustreError: 26552:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 261.864295] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 261.867693] Lustre: Skipped 3 previous similar messages [ 267.936803] Lustre: server umount lustre-MDT0001 complete [ 268.914545] Lustre: server umount lustre-OST0000 complete [ 269.907909] Lustre: server umount lustre-OST0001 complete [ 273.076201] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 275.326966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.200227] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 278.314581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 279.163465] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 279.910566] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 279.912381] Lustre: Skipped 2 previous similar messages [ 281.512946] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 281.517133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 282.780982] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 284.887629] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 284.892237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 286.176070] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 292.381123] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 302.968531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.969145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 302.969147] Lustre: Skipped 2 previous similar messages [ 302.977896] Lustre: Skipped 7 previous similar messages [ 305.455627] Lustre: server umount lustre-MDT0000 complete [ 306.422305] LustreError: 626:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718851941 with bad export cookie 2704434893257619266 [ 306.423681] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.428848] LustreError: 626:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 306.543527] Lustre: server umount lustre-MDT0001 complete [ 316.991983] Lustre: server umount lustre-OST0000 complete [ 327.971808] Lustre: server umount lustre-OST0001 complete [ 329.909108] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 22:52:45 (1718851965) [ 335.980532] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 338.205066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.326991] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.332191] LustreError: Skipped 5 previous similar messages [ 338.342646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 338.346199] Lustre: Skipped 7 previous similar messages [ 339.105380] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 341.313048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 342.192058] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 344.610253] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 344.613552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 345.895087] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 348.031378] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 348.034975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 349.278633] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 355.487849] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 356.348238] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 356.350684] Lustre: Skipped 2 previous similar messages [ 365.816928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.817318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.817320] Lustre: Skipped 3 previous similar messages [ 365.823772] Lustre: Skipped 3 previous similar messages [ 368.333686] Lustre: server umount lustre-MDT0000 complete [ 369.300100] LustreError: 7831:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718852004 with bad export cookie 2704434893257620372 [ 369.301636] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.307524] LustreError: 7831:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 369.404461] Lustre: server umount lustre-MDT0001 complete [ 379.837977] Lustre: server umount lustre-OST0000 complete [ 390.889127] Lustre: server umount lustre-OST0001 complete [ 392.890320] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 22:53:48 (1718852028) [ 395.912914] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 398.220839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.342435] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.348276] LustreError: Skipped 1 previous similar message [ 399.088308] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 401.287399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 402.223406] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 404.704369] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 404.707883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 406.017464] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 408.267296] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 408.271975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 409.557449] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 415.783436] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 426.345461] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.352093] Lustre: Skipped 3 previous similar messages [ 428.755765] Lustre: server umount lustre-MDT0000 complete [ 429.714275] LustreError: 15010:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718852065 with bad export cookie 2704434893257621478 [ 429.715256] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 429.719832] LustreError: 15010:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 430.158875] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 430.162453] Lustre: Skipped 8 previous similar messages [ 435.840623] Lustre: server umount lustre-MDT0001 complete [ 438.006031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.035530] LustreError: 18891:0:(import.c:353:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 440.040920] LustreError: 18891:0:(import.c:377:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff88009dd06a00 x1802346604672384/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1718852075 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_00_00.0' uid:0 gid:0 [ 440.052163] LustreError: 18891:0:(import.c:387:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 440.360871] LustreError: 3681:0:(client.c:1289:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88009dd04700 x1802346604673216/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 441.237486] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 443.449171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 444.334917] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 445.380417] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 460.553777] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 465.560006] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 465.564952] Lustre: Skipped 1 previous similar message [ 466.161837] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 20 sec [ 466.718184] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 467.094033] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 468.568642] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 474.052630] Lustre: server umount lustre-MDT0000 complete [ 475.014225] LustreError: 14114:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718852110 with bad export cookie 2704434893257622626 [ 475.015733] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.019795] LustreError: 14114:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 475.132523] Lustre: server umount lustre-MDT0001 complete [ 486.144577] Lustre: server umount lustre-OST0000 complete [ 497.147872] Lustre: server umount lustre-OST0001 complete [ 499.135111] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 22:55:34 (1718852134) [ 502.112377] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 504.329503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.471094] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.476014] LustreError: Skipped 8 previous similar messages [ 504.486257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 504.487883] Lustre: Skipped 9 previous similar messages [ 505.363863] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 507.574031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 508.438820] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 509.182722] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 509.184224] Lustre: Skipped 3 previous similar messages [ 511.091829] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 511.098555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 512.419241] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 514.579639] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 514.586080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 515.973790] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 522.296956] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 532.696128] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 532.698821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.703825] Lustre: Skipped 6 previous similar messages [ 537.699792] Lustre: server umount lustre-MDT0000 complete [ 538.767487] Lustre: server umount lustre-MDT0001 complete [ 550.013471] Lustre: server umount lustre-OST0000 complete [ 561.195871] Lustre: server umount lustre-OST0001 complete [ 563.139685] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 22:56:38 (1718852198) [ 563.698875] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 564.075298] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 22:56:39 (1718852199) [ 566.324602] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 567.627338] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 569.715869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 571.127991] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 571.134144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 572.839147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 572.861737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.952955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 572.991285] Lustre: lustre-MDT0000: new disk, initializing [ 573.013539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 573.755530] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 575.969778] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 577.625243] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 577.628561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 577.646601] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 577.649474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 578.915752] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 579.708548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 579.715537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 581.145153] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 581.612583] Lustre: DEBUG MARKER: create test files [ 589.720502] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.720649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.720651] Lustre: Skipped 14 previous similar messages [ 594.477587] Lustre: server umount lustre-OST0000 complete [ 595.658377] Lustre: server umount lustre-MDT0000 complete [ 598.435974] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 598.882769] Key type lgssc unregistered [ 598.947943] LNet: 768:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.952630] LNet: Removed LNI 192.168.202.133@tcp [ 607.357333] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 607.507702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 607.514759] alg: No test for adler32 (adler32-zlib) [ 608.322021] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 608.371021] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 608.372829] LNet: Accept secure, port 988 [ 609.905531] Key type lgssc registered [ 610.122120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 611.981284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 611.988514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.119159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.851759] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 614.846744] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 616.505524] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 616.508978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 616.611155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.795426] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 618.815740] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 623.611933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 623.629079] Lustre: 3752:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 623.631640] Lustre: 3752:0:(osd_handler.c:1974:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 623.633741] Lustre: 3752:0:(osd_handler.c:1981:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 623.635826] Lustre: 3752:0:(osd_handler.c:1991:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 623.638152] Lustre: 3752:0:(osd_handler.c:1998:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 623.640206] Lustre: 3752:0:(osd_handler.c:2005:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 626.539124] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 627.572328] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 633.624065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.624067] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 633.629516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 635.007773] Lustre: server umount lustre-OST0000 complete [ 636.072273] Lustre: server umount lustre-MDT0000 complete [ 638.889333] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 639.329151] Key type lgssc unregistered [ 639.391673] LNet: 5415:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.399682] LNet: Removed LNI 192.168.202.133@tcp [ 648.323743] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 648.435270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.459278] alg: No test for adler32 (adler32-zlib) [ 649.265705] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 649.310199] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 649.311889] LNet: Accept secure, port 988 [ 650.837489] Key type lgssc registered [ 651.032777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.876537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 652.883326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.018462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 654.749829] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 655.791464] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 657.493864] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 657.498494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 657.585138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.778329] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 659.783590] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 664.585162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 664.601507] Lustre: 8507:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 664.604527] Lustre: 8507:0:(osd_handler.c:1974:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 664.607665] Lustre: 8507:0:(osd_handler.c:1981:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 664.610365] Lustre: 8507:0:(osd_handler.c:1991:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 664.612682] Lustre: 8507:0:(osd_handler.c:1998:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 664.615209] Lustre: 8507:0:(osd_handler.c:2005:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 667.553854] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 669.592149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.592156] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 669.600886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 674.599985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 675.149136] Lustre: server umount lustre-OST0000 complete [ 676.179740] Lustre: server umount lustre-MDT0000 complete [ 678.997156] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 679.419887] Key type lgssc unregistered [ 679.480675] LNet: 10088:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.485693] LNet: Removed LNI 192.168.202.133@tcp [ 685.079300] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 689.456469] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 689.613014] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 689.619988] alg: No test for adler32 (adler32-zlib) [ 690.437970] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 690.494164] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 690.496486] LNet: Accept secure, port 988 [ 692.029486] Key type lgssc registered [ 692.218749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 694.229751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 695.973274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 697.605768] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 697.610062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 699.447598] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 699.452829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 701.343646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 701.362831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 701.366923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.440135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 702.447818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 702.486060] Lustre: lustre-MDT0000: new disk, initializing [ 702.503643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.508207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 703.182271] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 706.534683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 706.553578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 706.579632] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 706.594539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 706.596136] Lustre: Skipped 1 previous similar message [ 706.628783] Lustre: lustre-MDT0001: new disk, initializing [ 706.651680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 706.661212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 706.665085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 707.464777] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 709.780102] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 710.362623] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 712.110593] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 712.113554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 712.127955] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 712.130787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 712.204783] Lustre: lustre-OST0000: new disk, initializing [ 712.206297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 712.227238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.491789] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 715.219323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 715.223003] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 715.241533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 715.802580] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 716.581643] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 716.630893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 717.411604] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 717.460931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 720.247908] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 720.248579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.248581] Lustre: Skipped 1 previous similar message [ 720.248898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 720.255957] LustreError: Skipped 1 previous similar message [ 724.007147] Lustre: server umount lustre-OST0000 complete [ 725.255998] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.256409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.256411] Lustre: Skipped 1 previous similar message [ 725.264797] Lustre: Skipped 1 previous similar message [ 730.263901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 730.266594] Lustre: Skipped 2 previous similar messages [ 731.048175] Lustre: server umount lustre-MDT0000 complete [ 732.002001] LustreError: 13789:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718852367 with bad export cookie 9744184107967545210 [ 732.003080] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 732.006843] LustreError: 13789:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 732.091935] Lustre: server umount lustre-MDT0001 complete [ 734.037498] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 22:59:29 (1718852369) [ 735.720904] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 738.483092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 744.227573] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 745.686216] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 747.777140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 749.344773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 750.814903] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 750.819224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 752.358943] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 752.362996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 754.076009] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 754.089840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.154889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 754.161736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 754.193337] Lustre: lustre-MDT0000: new disk, initializing [ 754.210808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.215751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 754.948338] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 758.373189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 758.392099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 758.417497] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 758.430448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 758.432093] Lustre: Skipped 1 previous similar message [ 758.470049] Lustre: lustre-MDT0001: new disk, initializing [ 758.491968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 758.502404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 758.507155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 759.281205] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 761.529222] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 762.069869] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 763.743058] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 763.747391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 763.767424] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 763.771839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 765.120808] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 767.396080] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 768.201506] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 769.305191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 769.308957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 769.321223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 770.274786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 771.062067] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 771.110364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 774.328063] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 774.328267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.328549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 774.336520] LustreError: Skipped 1 previous similar message [ 777.666839] Lustre: server umount lustre-OST0000 complete [ 779.336004] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.336370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.336371] Lustre: Skipped 1 previous similar message [ 779.347244] Lustre: Skipped 2 previous similar messages [ 784.763699] Lustre: server umount lustre-MDT0000 complete [ 785.727101] LustreError: 26421:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718852421 with bad export cookie 9744184107967546050 [ 785.728403] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.733642] LustreError: 26421:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 785.832879] Lustre: server umount lustre-MDT0001 complete [ 787.808177] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 23:00:22 (1718852422) [ 789.491640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.619600] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 789.634510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.636639] Lustre: Skipped 1 previous similar message [ 790.386877] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 792.567788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 793.384963] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 794.437831] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 795.043718] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 796.882548] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 796.885691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 798.167118] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 799.197887] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 800.153473] Lustre: 30236:0:(genops.c:1674:obd_export_evict_by_uuid()) MGS: evicting e1ab69c0-97ce-4f56-bf9d-7b56dc55894a at adminstrative request [ 801.971625] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 801.977587] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 811.977992] Lustre: MGS: Client e1ab69c0-97ce-4f56-bf9d-7b56dc55894a (at 0@lo) reconnecting [ 811.980278] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x873a4bd1bd69860a to 0x873a4bd1bd698880 [ 811.983796] Lustre: MGC192.168.202.133@tcp: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 831.959908] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 831.962931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 831.976808] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 831.980903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 832.013463] Lustre: lustre-OST0001: new disk, initializing [ 832.015360] Lustre: Skipped 1 previous similar message [ 832.016821] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 832.018311] Lustre: Skipped 1 previous similar message [ 832.036220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 832.037967] Lustre: Skipped 2 previous similar messages [ 833.262790] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 835.668650] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 836.411205] Lustre: server umount lustre-OST0001 complete [ 839.032121] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 839.036268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.042624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 839.045213] Lustre: Skipped 3 previous similar messages [ 839.887280] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 843.523829] Lustre: server umount lustre-OST0000 complete [ 846.915032] Lustre: server umount lustre-MDT0000 complete [ 848.050280] LustreError: 32125:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718852483 with bad export cookie 9744184107967547520 [ 848.051459] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 848.056440] LustreError: 32125:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 848.177293] Lustre: server umount lustre-MDT0001 complete [ 850.298845] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 23:01:25 (1718852485) [ 850.698943] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 851.182700] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 23:01:26 (1718852486) [ 851.584578] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 852.101800] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 23:01:27 (1718852487) [ 852.564124] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 853.007086] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 23:01:28 (1718852488) [ 853.434663] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 854.903066] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 854.908190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 855.271154] loop: module loaded [ 855.278020] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 855.286952] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 855.827279] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 856.555654] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 857.570011] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 860.517864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 861.935488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 863.214467] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 863.220847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 864.653378] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 864.661388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 866.960957] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 866.968352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 866.996603] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 867.004763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 867.014418] LustreError: 3279:0:(obd_config.c:793:class_setup()) setup lustre-OST0000-osd failed (-28) [ 867.019750] LustreError: 3279:0:(obd_mount.c:214:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 867.023730] LustreError: 3279:0:(tgt_mount.c:2186:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 867.028153] LustreError: 3279:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 872.731472] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 874.643420] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 877.760488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 880.105002] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 882.159245] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 882.164456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 884.354266] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 884.357494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 886.568413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 886.584015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.662072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 886.670611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 886.708591] Lustre: lustre-MDT0000: new disk, initializing [ 886.725192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 886.729675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 887.651884] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 892.140897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 892.165794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 892.333321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 893.391248] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 895.823695] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 896.434086] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 898.763404] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 898.768668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 898.788506] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 898.793657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 900.329271] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 901.272537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 901.304451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 903.013026] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 904.188381] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 904.271883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 905.393343] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 905.475765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 906.296455] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 906.297481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.297484] Lustre: Skipped 3 previous similar messages [ 906.298313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 906.298315] Lustre: Skipped 4 previous similar messages [ 906.320986] LustreError: Skipped 1 previous similar message [ 912.176443] Lustre: server umount lustre-OST0000 complete [ 919.238065] Lustre: server umount lustre-MDT0000 complete [ 920.287179] LustreError: 6802:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718852555 with bad export cookie 9744184107967548066 [ 920.288187] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 920.294630] LustreError: 6802:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 920.399879] Lustre: server umount lustre-MDT0001 complete [ 922.545258] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 23:02:37 (1718852557) [ 924.573251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.741856] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.747298] LustreError: Skipped 1 previous similar message [ 925.540323] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 928.012543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 928.876464] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 929.842854] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 930.350370] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 932.305522] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 932.309768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 933.782882] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 934.799857] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 935.971570] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 936.056765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 936.940195] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 939.027931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 940.858110] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 940.861805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 940.878997] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 940.881870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 940.917148] Lustre: lustre-OST0001: new disk, initializing [ 940.918598] Lustre: Skipped 2 previous similar messages [ 940.920543] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 940.922754] Lustre: Skipped 3 previous similar messages [ 942.288416] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 944.681592] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 945.579246] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 947.459698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 947.462636] Lustre: Skipped 2 previous similar messages [ 947.463903] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 947.478882] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 947.664225] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 948.469905] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 948.522487] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 957.871629] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 963.470143] Lustre: Failing over lustre-MDT0000 [ 963.593166] Lustre: server umount lustre-MDT0000 complete [ 964.728218] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.737787] LustreError: Skipped 1 previous similar message [ 967.496701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.507134] Lustre: Skipped 5 previous similar messages [ 969.722591] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.731168] LustreError: Skipped 5 previous similar messages [ 977.322407] LDISKFS-fs (dm-0): recovery complete [ 977.324286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.352413] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 977.423245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 977.425140] Lustre: Skipped 6 previous similar messages [ 977.437336] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 978.268004] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 979.737185] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 979.737500] Lustre: 14077:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 982.424920] Lustre: 11565:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 982.433933] Lustre: 11565:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 1 previous similar message [ 982.440060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 982.459449] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 982.766490] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 983.174023] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 983.179859] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 1 previous similar message [ 983.484517] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 983.490734] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 1 previous similar message [ 983.496423] Lustre: 18179:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 983.502404] Lustre: 18179:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 7 previous similar messages [ 984.310526] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 984.315560] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 2 previous similar messages [ 984.619522] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 984.623163] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 2 previous similar messages [ 985.831596] Lustre: 18179:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 985.837254] Lustre: 18179:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 5 previous similar messages [ 986.641502] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 986.646520] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 4 previous similar messages [ 986.651304] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 986.656070] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 5 previous similar messages [ 990.171217] Lustre: 18179:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 990.174513] Lustre: 18179:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 10 previous similar messages [ 990.976480] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 990.978449] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 10 previous similar messages [ 990.982026] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 990.984364] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 10 previous similar messages [ 998.511172] Lustre: 18179:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 998.514555] Lustre: 18179:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 21 previous similar messages [ 999.316567] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 999.320939] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 21 previous similar messages [ 999.327339] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 999.331233] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 21 previous similar messages [ 1014.875231] Lustre: 18179:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1014.878870] Lustre: 18179:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 42 previous similar messages [ 1015.681540] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1015.688183] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 42 previous similar messages [ 1015.695458] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1015.701766] LustreError: 18179:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 42 previous similar messages [ 1039.775494] Lustre: 18179:0:(ldlm_lib.c:2300:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1039.777961] Lustre: 18179:0:(ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1039.780369] Lustre: 18179:0:(genops.c:1531:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client fa164b72-9294-4435-95d8-4d1b6215bfaa@192.168.202.33@tcp [ 1039.785166] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1039.787212] Lustre: 18179:0:(ldlm_lib.c:1824:abort_req_replay_queue()) @@@ aborted: req@ffff8800a1873800 x1802347291373888/t0(8589934747) o101->fa164b72-9294-4435-95d8-4d1b6215bfaa@192.168.202.33@tcp:55/0 lens 592/0 e 0 to 0 dl 1718852685 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1039.795788] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1039.800198] Lustre: lustre-MDT0000: Denying connection for new client fa164b72-9294-4435-95d8-4d1b6215bfaa (at 192.168.202.33@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1039.803150] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1039.808866] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 1039.808917] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1039.814176] Lustre: Skipped 2 previous similar messages [ 1039.823393] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:39 to 0x2c0000401:545) [ 1039.823408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:72 to 0x280000401:577) [ 1042.522209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.522952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1042.522955] Lustre: Skipped 5 previous similar messages [ 1042.535472] Lustre: Skipped 2 previous similar messages [ 1048.013124] Lustre: server umount lustre-OST0000 complete [ 1049.960462] LustreError: 0-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1052.536293] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.539887] LustreError: Skipped 7 previous similar messages [ 1055.379082] Lustre: server umount lustre-OST0001 complete [ 1062.451483] Lustre: server umount lustre-MDT0000 complete [ 1063.423475] LustreError: 11529:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718852698 with bad export cookie 9744184107967599152 [ 1063.423824] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1063.430222] LustreError: 11529:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1063.529334] Lustre: server umount lustre-MDT0001 complete [ 1065.479909] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 23:05:00 (1718852700) [ 1067.841120] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1067.846971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1067.852145] Lustre: *** cfs_fail_loc=197, val=0*** [ 1084.946939] LustreError: 13a-8: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1100.950905] LustreError: 20726:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 1106.964588] Lustre: *** cfs_fail_loc=197, val=0*** [ 1106.965710] Lustre: Skipped 65 previous similar messages [ 1129.980627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.133503] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1132.266199] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1133.065163] Lustre: server umount lustre-OST0000 complete [ 1135.110138] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 23:06:10 (1718852770) [ 1147.886173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.020275] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.024418] LustreError: Skipped 2 previous similar messages [ 1148.742136] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1151.018779] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1152.816456] LustreError: 22618:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1152.824592] LustreError: 22618:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1152.828819] LustreError: 22618:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1152.952896] Lustre: server umount lustre-MDT0000 complete [ 1157.817269] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1157.822298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1162.834646] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 1164.675857] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1167.013777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1169.220774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1170.728528] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1170.734161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1172.691026] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1172.694154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1174.376535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1174.393402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.471967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1174.474783] Lustre: Skipped 1 previous similar message [ 1174.482090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1174.521333] Lustre: lustre-MDT0000: new disk, initializing [ 1174.544070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1175.297872] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1179.277742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1179.301594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1179.428724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1180.402253] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1183.128161] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1183.876722] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1186.034456] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1186.041640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1186.063297] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1186.069983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1186.370274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1187.356085] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1190.050807] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1191.191952] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1191.276918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1192.453259] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1192.532815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1195.350555] Lustre: server umount lustre-OST0000 complete [ 1199.432672] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1199.437178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.444717] Lustre: Skipped 4 previous similar messages [ 1199.448230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.452957] Lustre: Skipped 8 previous similar messages [ 1206.404221] Lustre: server umount lustre-MDT0000 complete [ 1207.767733] LustreError: 27565:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718852843 with bad export cookie 9744184107967609456 [ 1207.770141] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.781785] LustreError: 27565:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1217.865823] Lustre: server umount lustre-MDT0001 complete [ 1220.451132] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 23:07:35 (1718852855) [ 1223.509809] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1224.268170] Key type lgssc unregistered [ 1224.357958] LNet: 32216:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.365910] LNet: Removed LNI 192.168.202.133@tcp [ 1231.450953] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 1236.823058] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1237.015977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1237.019453] alg: No test for adler32 (adler32-zlib) [ 1237.830137] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 1237.892127] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1237.894471] LNet: Accept secure, port 988 [ 1239.435674] Key type lgssc registered [ 1239.785116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.966497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1245.329193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1247.399303] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1247.405519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1249.685160] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1249.691629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1251.769192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1254.253075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1254.277054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1254.286796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.373047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1255.380154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1255.437435] Lustre: lustre-MDT0000: new disk, initializing [ 1255.464574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.472991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1256.611280] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1260.939702] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1260.948392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1260.974810] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1260.980048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1261.130580] Lustre: lustre-OST0000: new disk, initializing [ 1261.134920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1261.139060] Lustre: Skipped 1 previous similar message [ 1261.170308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.072051] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1263.793689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1263.799398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1263.811582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1267.530213] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1267.536063] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1267.555962] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1267.561609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1267.619460] Lustre: lustre-OST0001: new disk, initializing [ 1267.623225] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1267.652236] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1269.277797] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1272.794851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1272.800860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1272.815256] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1273.433380] Lustre: server umount lustre-MDT0000 complete [ 1275.519853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1277.263703] LustreError: 6413:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718852912 with bad export cookie 1567255098005788525 [ 1277.270820] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.302915] Lustre: server umount lustre-OST0000 complete [ 1298.804095] Lustre: server umount lustre-OST0001 complete [ 1301.953059] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 23:08:56 (1718852936) [ 1304.267713] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1306.878696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1307.264930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1309.239094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1309.633515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1311.219540] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 23:09:06 (1718852946) [ 1315.860485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1316.581872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1317.350472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1318.075910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1318.457819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1319.212673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1325.015900] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 1327.337028] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1330.386436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1332.708776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1334.684720] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1334.690302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1336.895195] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1336.900908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1339.347227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1339.370805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.472598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1339.482191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1339.524946] Lustre: lustre-MDT0000: new disk, initializing [ 1339.545910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1339.551504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1340.212937] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1344.483212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1344.506380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1344.548571] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1344.572474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1344.576694] Lustre: Skipped 1 previous similar message [ 1344.629334] Lustre: lustre-MDT0001: new disk, initializing [ 1344.666519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1344.681727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1344.687664] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1345.704130] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1348.259221] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1349.022006] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1351.240657] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1351.243958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1351.262454] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1351.267198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1352.776017] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1355.257728] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1355.700247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1355.707170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1355.738536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1356.334988] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1356.416094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1357.473328] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1357.550184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1360.728549] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1360.729011] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.729554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1364.339375] Lustre: server umount lustre-OST0000 complete [ 1365.736219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.736722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1365.736724] Lustre: Skipped 1 previous similar message [ 1365.754786] Lustre: Skipped 2 previous similar messages [ 1370.744150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1370.747703] Lustre: Skipped 2 previous similar messages [ 1371.781015] Lustre: server umount lustre-MDT0000 complete [ 1373.178750] LustreError: 17810:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718853008 with bad export cookie 1567255098005789701 [ 1373.181074] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1373.194196] LustreError: 17810:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1373.341533] Lustre: server umount lustre-MDT0001 complete [ 1376.292167] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 23:10:11 (1718853011) [ 1378.689113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.852806] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.870696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.874306] Lustre: Skipped 1 previous similar message [ 1380.021473] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1383.133322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1384.345196] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1385.779754] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1386.597606] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1388.425055] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1388.430383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1389.940877] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1391.357605] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1397.022779] Lustre: *** cfs_fail_loc=159, val=0*** [ 1413.033425] Lustre: lustre-MDT0000: Client 86b72691-9ab1-4c26-a15d-9bc137b1d7f8 (at 192.168.202.33@tcp) reconnecting [ 1413.333048] Lustre: *** cfs_fail_loc=159, val=0*** [ 1413.334313] Lustre: Skipped 6 previous similar messages [ 1429.333779] Lustre: lustre-MDT0000: Client 86b72691-9ab1-4c26-a15d-9bc137b1d7f8 (at 192.168.202.33@tcp) reconnecting [ 1433.608288] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1433.608362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.608975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1433.621599] LustreError: Skipped 1 previous similar message [ 1436.186123] Lustre: server umount lustre-OST0000 complete [ 1438.344400] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1438.348386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.355070] Lustre: Skipped 1 previous similar message [ 1438.357864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1438.361064] Lustre: Skipped 1 previous similar message [ 1443.449777] Lustre: server umount lustre-MDT0000 complete [ 1443.623969] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.631139] LustreError: Skipped 1 previous similar message [ 1444.797506] LustreError: 18945:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718853080 with bad export cookie 1567255098005790541 [ 1444.799798] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1444.814228] LustreError: 18945:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1444.957792] Lustre: server umount lustre-MDT0001 complete [ 1448.707222] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1449.491524] Key type lgssc unregistered [ 1449.589978] LNet: 22870:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.601993] LNet: Removed LNI 192.168.202.133@tcp [ 1458.474096] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1458.628129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1458.637672] alg: No test for adler32 (adler32-zlib) [ 1459.443866] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 1459.491481] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1459.493544] LNet: Accept secure, port 988 [ 1461.024682] Key type lgssc registered [ 1461.371075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1463.745285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1463.753069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.903014] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.918747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1466.089163] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1469.227116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1469.379834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1470.538315] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1471.971936] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1472.689511] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1475.182760] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1475.188748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1475.351909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.350670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 1476.975348] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1478.169315] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1487.114890] Lustre: *** cfs_fail_loc=159, val=0*** [ 1503.137887] Lustre: lustre-MDT0000: Client 9c4eef1a-4efc-4b19-9ec9-3e354e968bff (at 192.168.202.33@tcp) reconnecting [ 1503.692789] Lustre: *** cfs_fail_loc=159, val=0*** [ 1519.703130] Lustre: lustre-MDT0001: Client 9c4eef1a-4efc-4b19-9ec9-3e354e968bff (at 192.168.202.33@tcp) reconnecting [ 1520.079212] Lustre: *** cfs_fail_loc=159, val=0*** [ 1520.081108] Lustre: Skipped 4 previous similar messages [ 1536.080974] Lustre: lustre-MDT0001: Client 9c4eef1a-4efc-4b19-9ec9-3e354e968bff (at 192.168.202.33@tcp) reconnecting [ 1536.779906] Lustre: 27491:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1537.378552] Lustre: *** cfs_fail_loc=159, val=0*** [ 1537.381134] Lustre: Skipped 3 previous similar messages [ 1553.384429] Lustre: lustre-MDT0000: Client ea688309-8977-4c3c-afc9-a343c4efbdd0 (at 192.168.202.33@tcp) reconnecting [ 1553.728248] Lustre: *** cfs_fail_loc=159, val=0*** [ 1553.731928] Lustre: Skipped 13 previous similar messages [ 1569.732410] Lustre: lustre-MDT0000: Client ea688309-8977-4c3c-afc9-a343c4efbdd0 (at 192.168.202.33@tcp) reconnecting [ 1570.008666] Lustre: 27840:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1571.512758] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1571.517894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.526008] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1574.505035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.505273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1574.517045] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1577.139872] Lustre: server umount lustre-OST0000 complete [ 1579.512702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.522230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.526386] Lustre: Skipped 2 previous similar messages [ 1584.520153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1584.523369] Lustre: Skipped 1 previous similar message [ 1584.553559] Lustre: server umount lustre-MDT0000 complete [ 1585.839788] LustreError: 26229:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718853221 with bad export cookie 17464071913045301601 [ 1585.841712] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1585.853171] LustreError: 26229:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1585.985288] Lustre: server umount lustre-MDT0001 complete [ 1589.667817] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1590.340515] Key type lgssc unregistered [ 1590.408855] LNet: 29042:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.414898] LNet: Removed LNI 192.168.202.133@tcp [ 1599.643216] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1599.803549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1599.808425] alg: No test for adler32 (adler32-zlib) [ 1600.617273] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 1600.668858] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1600.670554] LNet: Accept secure, port 988 [ 1602.201605] Key type lgssc registered [ 1602.574544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1605.525451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1605.534821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.689380] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.701407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.623568] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1610.800925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1610.968213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1612.075522] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1613.481603] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1614.277707] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1616.709148] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1616.715878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1616.859374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.862386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 1618.697259] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1620.283584] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1620.864327] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 1623.442977] Lustre: 655:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1624.017678] Lustre: 721:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1624.545287] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1625.564974] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1625.567379] Lustre: Skipped 1 previous similar message [ 1626.163634] Lustre: 1076:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1627.880293] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1627.885274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1627.892946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1630.872989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1630.873133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.884697] Lustre: Skipped 1 previous similar message [ 1633.094925] Lustre: server umount lustre-OST0000 complete [ 1635.880008] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.884615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1640.234253] Lustre: server umount lustre-MDT0000 complete [ 1640.888067] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.894290] LustreError: Skipped 1 previous similar message [ 1641.294195] LustreError: 1098:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718853276 with bad export cookie 7175302840031424282 [ 1641.296055] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1641.308907] LustreError: 1098:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1641.423521] Lustre: server umount lustre-MDT0001 complete [ 1644.476571] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1645.094998] Key type lgssc unregistered [ 1645.163798] LNet: 2284:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.170172] LNet: Removed LNI 192.168.202.133@tcp [ 1652.492390] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1652.580337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1652.587053] alg: No test for adler32 (adler32-zlib) [ 1653.395497] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 1653.444460] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1653.446258] LNet: Accept secure, port 988 [ 1654.971541] Key type lgssc registered [ 1655.176868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1657.107047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1657.113241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.265340] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1658.276756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.072987] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1661.207254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1661.309531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1662.020735] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1662.992328] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1663.516467] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1665.250890] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1665.254832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1665.346270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.513668] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1667.505266] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1668.345922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 1671.346635] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 1675.360331] Lustre: *** cfs_fail_loc=159, val=0*** [ 1691.364860] Lustre: lustre-MDT0000: Client 6c5994de-c77b-4121-ae02-d4e719b65411 (at 192.168.202.33@tcp) reconnecting [ 1693.384048] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1693.386095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.389164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1696.392093] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1696.392276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1696.396550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.181072] Lustre: server umount lustre-OST0000 complete [ 1701.351992] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1701.354795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.359370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.361206] Lustre: Skipped 1 previous similar message [ 1705.194261] Lustre: server umount lustre-MDT0000 complete [ 1706.130140] LustreError: 3773:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718853341 with bad export cookie 16069171675506157278 [ 1706.131240] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1706.136267] LustreError: 3773:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1706.228792] Lustre: server umount lustre-MDT0001 complete [ 1709.032870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1709.496259] Key type lgssc unregistered [ 1709.563867] LNet: 7575:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.569695] LNet: Removed LNI 192.168.202.133@tcp [ 1716.792192] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1716.956765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.962285] alg: No test for adler32 (adler32-zlib) [ 1717.777070] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 1717.830294] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1717.832716] LNet: Accept secure, port 988 [ 1719.363535] Key type lgssc registered [ 1719.575773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1721.514284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1721.519756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.650490] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.664568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.494224] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1725.741470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1725.852217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1726.677571] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1727.746151] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1728.360030] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1730.208885] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1730.212797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1730.315025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1731.641087] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1732.752923] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1733.316104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 1736.314876] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 1750.543548] Lustre: lustre-OST0000: evicting 57899952-192d-4f6f-a2eb-2e6d05b87a51 (at 192.168.202.33@tcp) by administrative request [ 1750.546148] Lustre: Skipped 1 previous similar message [ 1776.376266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.376493] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1776.376667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.384150] Lustre: Skipped 1 previous similar message [ 1779.992713] Lustre: server umount lustre-OST0000 complete [ 1781.384203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.390057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.392127] Lustre: Skipped 1 previous similar message [ 1785.927918] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1785.930605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.936136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.992515] Lustre: server umount lustre-MDT0000 complete [ 1787.938233] LustreError: 12594:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718853423 with bad export cookie 379380580137124837 [ 1787.940422] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1787.946163] LustreError: 12594:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1788.047178] Lustre: server umount lustre-MDT0001 complete [ 1790.750405] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1791.136645] Key type lgssc unregistered [ 1791.195671] LNet: 13389:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.200642] LNet: Removed LNI 192.168.202.133@tcp [ 1809.385240] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1809.504280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1809.507915] alg: No test for adler32 (adler32-zlib) [ 1810.318255] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 1810.373296] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1810.375530] LNet: Accept secure, port 988 [ 1811.915575] Key type lgssc registered [ 1812.247294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1814.576072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1814.584060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.706961] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.717077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.561118] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1818.635251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1818.724609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1819.548127] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1820.531391] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1821.080166] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1822.983284] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1822.986508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1823.074898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.478718] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1825.516643] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1826.075875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1829.076485] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 1834.088054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.088390] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1834.088499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1834.093995] Lustre: Skipped 1 previous similar message [ 1835.813884] Lustre: server umount lustre-OST0000 complete [ 1838.744073] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1838.747000] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.751239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.753144] Lustre: Skipped 1 previous similar message [ 1842.797560] Lustre: server umount lustre-MDT0000 complete [ 1844.104241] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1844.112316] LustreError: Skipped 1 previous similar message [ 1844.194574] LustreError: 16349:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718853479 with bad export cookie 14609457097262929719 [ 1844.196987] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1844.206453] LustreError: 16349:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1844.337127] Lustre: server umount lustre-MDT0001 complete [ 1847.869573] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1848.653301] Key type lgssc unregistered [ 1848.710748] LNet: 19954:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.715898] LNet: Removed LNI 192.168.202.133@tcp [ 1857.729965] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1857.911243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1857.916433] alg: No test for adler32 (adler32-zlib) [ 1858.725990] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 1858.775361] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1858.777991] LNet: Accept secure, port 988 [ 1860.307557] Key type lgssc registered [ 1860.487143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1861.957713] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1861.961750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1863.553393] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 23:18:18 (1718853498) [ 1867.001713] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 1868.351844] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1870.289207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1871.824160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1873.596758] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1873.599867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1875.004220] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1875.006980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1876.650408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1876.666406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1876.670551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.740497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1877.748492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1877.779748] Lustre: lustre-MDT0000: new disk, initializing [ 1877.795582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.800734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1878.506496] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1882.242974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1882.267940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1882.312000] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1882.333655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1882.336923] Lustre: Skipped 1 previous similar message [ 1882.391170] Lustre: lustre-MDT0001: new disk, initializing [ 1882.427285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1882.443247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1882.448769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1883.221012] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1885.893276] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1886.660480] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1888.202376] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1888.205181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1888.224030] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1888.227014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1888.295291] Lustre: lustre-OST0000: new disk, initializing [ 1888.297879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1888.314522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.383000] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1891.484605] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1895.698886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1895.701005] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1895.717107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1900.711849] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1900.713907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.717301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.886777] Lustre: server umount lustre-OST0000 complete [ 1907.448012] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1907.449964] LustreError: Skipped 1 previous similar message [ 1907.451025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.454079] Lustre: Skipped 1 previous similar message [ 1907.455414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.456891] Lustre: Skipped 1 previous similar message [ 1910.743916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.744113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.750912] Lustre: Skipped 1 previous similar message [ 1911.842880] Lustre: server umount lustre-MDT0000 complete [ 1912.733102] LustreError: 26965:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718853548 with bad export cookie 2996653055438808445 [ 1912.734064] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1912.740862] LustreError: 26965:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1912.833840] Lustre: server umount lustre-MDT0001 complete [ 1915.474550] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1915.873014] Key type lgssc unregistered [ 1915.929695] LNet: 29012:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.934666] LNet: Removed LNI 192.168.202.133@tcp [ 1922.841336] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1922.985022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1922.988544] alg: No test for adler32 (adler32-zlib) [ 1923.787423] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 1923.827951] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1923.829281] LNet: Accept secure, port 988 [ 1925.352634] Key type lgssc registered [ 1925.684004] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1928.307806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1928.311987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.477553] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1929.496663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.644740] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1933.639855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1933.765814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1934.815827] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1936.296208] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1937.001678] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1938.623155] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1938.626190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1938.710974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.725475] mount.lustre (32347) used greatest stack depth: 9392 bytes left [ 1939.712461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 1939.853598] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1940.946221] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1947.848227] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1947.855098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.860815] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1947.863688] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1947.868599] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 1948.464646] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1948.470349] Lustre: Skipped 1 previous similar message [ 1948.478489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.488298] Lustre: Skipped 1 previous similar message [ 1948.491926] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1948.497623] Lustre: Skipped 1 previous similar message [ 1948.501303] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1948.506994] LustreError: Skipped 1 previous similar message [ 1948.509990] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 1948.515313] Lustre: Skipped 1 previous similar message [ 1949.486898] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1949.491981] Lustre: Skipped 2 previous similar messages [ 1949.500073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.509925] Lustre: Skipped 2 previous similar messages [ 1949.514529] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1949.518615] Lustre: Skipped 2 previous similar messages [ 1949.521742] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1949.528111] LustreError: Skipped 2 previous similar messages [ 1949.530945] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 1949.538230] Lustre: Skipped 2 previous similar messages [ 1951.554147] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1951.559847] Lustre: Skipped 5 previous similar messages [ 1951.568264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.575337] Lustre: Skipped 5 previous similar messages [ 1951.578514] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1951.582472] Lustre: Skipped 5 previous similar messages [ 1951.585066] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1951.591419] LustreError: Skipped 5 previous similar messages [ 1951.594396] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 1951.598764] Lustre: Skipped 5 previous similar messages [ 1955.878491] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1955.882202] Lustre: Skipped 12 previous similar messages [ 1955.889189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1955.896190] Lustre: Skipped 12 previous similar messages [ 1955.898903] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1955.903392] Lustre: Skipped 12 previous similar messages [ 1955.906549] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1955.912712] LustreError: Skipped 12 previous similar messages [ 1955.915581] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 1955.920232] Lustre: Skipped 12 previous similar messages [ 1964.001589] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1964.003307] Lustre: Skipped 25 previous similar messages [ 1964.007681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.011420] Lustre: Skipped 25 previous similar messages [ 1964.013118] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1964.015230] Lustre: Skipped 25 previous similar messages [ 1964.016472] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1964.019022] LustreError: Skipped 25 previous similar messages [ 1964.020309] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 1964.022254] Lustre: Skipped 25 previous similar messages [ 1980.205191] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1980.207614] Lustre: Skipped 68 previous similar messages [ 1980.212087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.216044] Lustre: Skipped 68 previous similar messages [ 1980.217708] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1980.220853] Lustre: Skipped 68 previous similar messages [ 1980.222516] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1980.226772] LustreError: Skipped 68 previous similar messages [ 1980.229167] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 1980.232755] Lustre: Skipped 68 previous similar messages [ 2012.390146] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2012.391842] Lustre: Skipped 114 previous similar messages [ 2012.395150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.398050] Lustre: Skipped 114 previous similar messages [ 2012.399306] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2012.401146] Lustre: Skipped 114 previous similar messages [ 2012.402281] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2012.405537] LustreError: Skipped 114 previous similar messages [ 2012.407469] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 2012.410540] Lustre: Skipped 114 previous similar messages [ 2032.057922] LustreError: 30536:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2032.064527] LustreError: 30536:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2076.631494] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2076.635393] Lustre: Skipped 242 previous similar messages [ 2076.642848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.649651] Lustre: Skipped 242 previous similar messages [ 2076.653042] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2076.657726] Lustre: Skipped 242 previous similar messages [ 2076.660594] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2076.669088] LustreError: Skipped 242 previous similar messages [ 2076.672970] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 2076.677600] Lustre: Skipped 242 previous similar messages [ 2102.236996] LustreError: 30536:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2102.242893] LustreError: 30536:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2117.925266] LustreError: 30536:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2117.931327] LustreError: 30536:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2131.144336] LustreError: 30536:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2131.151103] LustreError: 30536:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2144.614521] LustreError: 30536:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2144.617885] LustreError: 30536:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2171.891293] LustreError: 30536:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2171.895258] LustreError: 30536:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2204.917492] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2204.921285] Lustre: Skipped 507 previous similar messages [ 2204.928401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.935426] Lustre: Skipped 507 previous similar messages [ 2204.938144] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2204.942236] Lustre: Skipped 507 previous similar messages [ 2204.945551] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2204.956106] LustreError: Skipped 507 previous similar messages [ 2204.961248] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 2204.966293] Lustre: Skipped 507 previous similar messages [ 2293.113721] LustreError: 32353:0:(ofd_dev.c:1695:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 2293.120075] LustreError: 30536:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 2293.128015] LustreError: 30536:0:(osp_precreate.c:674:osp_precreate_send()) Skipped 2 previous similar messages [ 2354.376238] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2354.376605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.383876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.385770] Lustre: Skipped 2 previous similar messages [ 2360.379540] Lustre: server umount lustre-OST0000 complete [ 2364.392317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2364.394188] Lustre: Skipped 1 previous similar message [ 2367.486607] Lustre: server umount lustre-MDT0000 complete [ 2368.762674] LustreError: 30499:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854004 with bad export cookie 8034570031039997719 [ 2368.764786] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2368.772870] LustreError: 30499:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2368.898382] Lustre: server umount lustre-MDT0001 complete [ 2371.694968] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2372.073524] Key type lgssc unregistered [ 2372.128673] LNet: 5031:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.133684] LNet: Removed LNI 192.168.202.133@tcp [ 2379.424849] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2379.550658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2379.557562] alg: No test for adler32 (adler32-zlib) [ 2380.365022] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 2380.416963] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2380.418562] LNet: Accept secure, port 988 [ 2381.953534] Key type lgssc registered [ 2382.119882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2384.032546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2384.039670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.176966] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.187772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.135768] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2388.199392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2388.302896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2389.050172] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2390.042641] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2390.573707] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2392.315174] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2392.319139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2392.432975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.781121] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2394.958569] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2395.439632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49666 to 0x280000401:49697) [ 2402.525818] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2402.967708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.971550] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2402.973595] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2402.976966] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 2402.976985] LustreError: 6555:0:(osp_precreate.c:1286:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2405.545165] Lustre: server umount lustre-OST0000 complete [ 2423.076467] LustreError: 6555:0:(osp_precreate.c:1286:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 2424.479491] Lustre: 5776:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1718854043/real 1718854043] req@ffff88007ab11880 x1802349068087744/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1718854059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 2424.479578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.499066] Lustre: 5776:0:(client.c:2340:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2429.479514] Lustre: 5775:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1718854048/real 1718854048] req@ffff880086bf9c00 x1802349068088384/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1718854064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 2429.493448] Lustre: 5775:0:(client.c:2340:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2433.681793] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2433.687063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2433.820719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.819126] LustreError: 0-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2434.826642] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 2434.831517] Lustre: Skipped 1 previous similar message [ 2435.639740] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2437.164193] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2447.891476] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 2448.800575] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff880136956800.ost_server_uuid 50 [ 2449.360449] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136956800.ost_server_uuid in FULL state after 0 sec [ 2454.856863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2454.857032] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2454.857571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2454.872808] Lustre: Skipped 2 previous similar messages [ 2456.407003] Lustre: server umount lustre-OST0000 complete [ 2458.392469] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2458.399966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.411516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2458.416929] Lustre: Skipped 1 previous similar message [ 2459.864389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2459.867835] Lustre: Skipped 1 previous similar message [ 2463.381371] Lustre: server umount lustre-MDT0000 complete [ 2464.494027] LustreError: 10730:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854099 with bad export cookie 8388072509264151306 [ 2464.495217] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2464.499184] LustreError: 10730:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2464.593163] Lustre: server umount lustre-MDT0001 complete [ 2467.564591] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2468.104909] Key type lgssc unregistered [ 2468.158768] LNet: 11905:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.163890] LNet: Removed LNI 192.168.202.133@tcp [ 2478.190542] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2478.358332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2478.365066] alg: No test for adler32 (adler32-zlib) [ 2479.173631] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 2479.225422] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2479.227259] LNet: Accept secure, port 988 [ 2480.760599] Key type lgssc registered [ 2481.088175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2483.869417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2483.879768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.025779] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.035892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.966286] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2489.087978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2489.198789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2490.227961] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2491.212707] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 2493.604612] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2493.609767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2493.740670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.741372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49729) [ 2495.551144] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2498.508060] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2498.510665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2498.525419] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2498.527970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2498.562998] Lustre: lustre-OST0001: new disk, initializing [ 2498.564665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2498.580810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2499.864893] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2504.904105] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2505.937390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2505.942648] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2505.976694] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2511.279387] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 2524.232196] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2524.237688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.245527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2525.992430] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.992498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2525.998385] Lustre: Skipped 2 previous similar messages [ 2528.966751] Lustre: server umount lustre-MDT0000 complete [ 2529.992534] LustreError: 17741:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854165 with bad export cookie 10218445132457772579 [ 2529.993561] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.994014] LustreError: 14693:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200001b72:0x1:0x0]: rc = -5 [ 2529.994111] LustreError: 14693:0:(client.c:1279:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880095739c00 x1802349171482368/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'dist_txn-1.0' uid:0 gid:0 [ 2530.006552] LustreError: 17741:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2530.098707] Lustre: server umount lustre-MDT0001 complete [ 2541.269044] Lustre: server umount lustre-OST0000 complete [ 2552.238140] Lustre: server umount lustre-OST0001 complete [ 2553.382465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2553.405325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2553.782545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2553.806395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2554.226787] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2554.235645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2554.256741] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2554.262341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2554.658258] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2554.665204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2554.692887] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2554.698603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2558.462238] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2560.976689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2561.108486] LustreError: 137-5: mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.124114] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.082462] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2564.384341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2565.153655] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2565.822017] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 2567.336782] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2567.340038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2568.540926] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2570.510958] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2570.514474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2570.568486] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2570.570490] Lustre: Skipped 2 previous similar messages [ 2571.568582] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49761) [ 2571.890842] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2574.568402] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 2578.042752] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2578.931205] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2589.496203] LustreError: 0-0: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2589.498609] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.501697] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2589.504115] Lustre: Skipped 3 previous similar messages [ 2593.555909] Lustre: server umount mylustre-MDT0000 complete [ 2594.462290] LustreError: 24048:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854229 with bad export cookie 10218445132457775386 [ 2594.463309] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2594.468784] LustreError: 24048:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2594.565033] Lustre: server umount mylustre-MDT0001 complete [ 2605.630856] Lustre: server umount mylustre-OST0000 complete [ 2616.679832] Lustre: server umount mylustre-OST0001 complete [ 2617.539165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2617.562633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2617.817028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2617.836376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2618.106982] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2618.110697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2618.133175] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2618.136728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2618.424824] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2618.427858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2618.446601] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2618.451372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2621.802405] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2624.071208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.201306] LustreError: 137-5: tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.215276] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.173600] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2627.370112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2628.109072] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2628.788887] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 2630.415719] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2630.418258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2631.810865] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2634.080059] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2634.085389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2635.171137] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49793) [ 2635.657825] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2637.170496] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2642.058756] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2642.925232] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2652.200653] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.201132] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.201135] Lustre: Skipped 4 previous similar messages [ 2652.207674] Lustre: Skipped 5 previous similar messages [ 2655.425390] Lustre: server umount tfs-MDT0000 complete [ 2656.546800] LustreError: 26929:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854291 with bad export cookie 10218445132457777339 [ 2656.548890] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2656.561468] LustreError: 26929:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2656.680662] Lustre: server umount tfs-MDT0001 complete [ 2667.914266] Lustre: server umount tfs-OST0000 complete [ 2679.130846] Lustre: server umount tfs-OST0001 complete [ 2680.269943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2680.292124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2680.582367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2680.606150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2680.992423] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2681.001702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2681.029915] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2681.036988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2681.403666] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2681.409606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2681.429952] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2681.434714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2684.738272] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2686.959788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.082086] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.100275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2687.101948] Lustre: Skipped 3 previous similar messages [ 2687.840853] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2689.876976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2690.699650] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2691.422256] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 2693.012566] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2693.015241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2694.338092] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2696.438560] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2696.441377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2697.486510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49825) [ 2697.601645] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2700.488878] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 2703.713793] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2711.276407] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 23:32:26 (1718854346) [ 2725.016278] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2725.019246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.024212] Lustre: Skipped 1 previous similar message [ 2725.026818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2725.031050] Lustre: Skipped 3 previous similar messages [ 2728.307967] Lustre: server umount lustre-MDT0000 complete [ 2729.240165] LustreError: 928:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854364 with bad export cookie 10218445132457779033 [ 2729.241133] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2729.245087] LustreError: 928:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2730.536016] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.540584] LustreError: Skipped 1 previous similar message [ 2734.543907] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2734.547776] Lustre: Skipped 6 previous similar messages [ 2735.341225] Lustre: server umount lustre-MDT0001 complete [ 2742.646940] Lustre: server umount lustre-OST0000 complete [ 2750.089171] Lustre: server umount lustre-OST0001 complete [ 2751.866860] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 2754.092908] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2757.204003] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2759.557394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2761.691420] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2761.700802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2763.971537] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2763.978408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2766.327651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2766.339627] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2766.343879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2766.405410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2766.407057] Lustre: Skipped 1 previous similar message [ 2766.413261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2766.444734] Lustre: lustre-MDT0000: new disk, initializing [ 2766.461145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2766.464167] Lustre: Skipped 3 previous similar messages [ 2766.469596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2767.282944] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2770.614675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.634180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2770.668397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2770.669938] Lustre: Skipped 1 previous similar message [ 2770.702331] Lustre: lustre-MDT0001: new disk, initializing [ 2770.733722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2770.736814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2771.640682] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2775.461660] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2775.464844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2775.481184] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2775.483774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2775.549801] Lustre: lustre-OST0000: new disk, initializing [ 2775.551405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2777.067152] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2777.294209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2777.300910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2777.329773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2781.061774] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2781.067160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2781.092645] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2781.097995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2781.157104] Lustre: lustre-OST0001: new disk, initializing [ 2781.160242] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2782.324804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2782.331723] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2782.356523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2783.035523] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2787.623236] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 23:33:42 (1718854422) [ 2790.729730] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 23:33:45 (1718854425) [ 2802.360439] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.360618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.360620] Lustre: Skipped 5 previous similar messages [ 2802.361143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.384936] LustreError: Skipped 1 previous similar message [ 2807.442029] Lustre: server umount lustre-OST0000 complete [ 2810.775976] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2812.392022] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.401225] LustreError: Skipped 2 previous similar messages [ 2814.430914] Lustre: server umount lustre-MDT0000 complete [ 2815.379079] LustreError: 10391:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854450 with bad export cookie 10218445132457780587 [ 2815.380170] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2815.384694] LustreError: 10391:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2821.477614] Lustre: server umount lustre-MDT0001 complete [ 2824.226895] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2832.883231] Lustre: server umount lustre-OST0001 complete [ 2834.084719] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 2838.064258] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2839.943889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2841.366545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2842.722310] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2842.726396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2844.150018] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2844.153124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2845.774741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2845.795249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2845.863270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2845.866004] Lustre: Skipped 1 previous similar message [ 2845.873210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2845.912654] Lustre: lustre-MDT0000: new disk, initializing [ 2845.935885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2846.664620] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2849.883324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2849.897138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2849.983045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2850.659828] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2852.797138] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2853.328905] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2854.906631] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2854.910020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2854.930100] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2854.933136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2856.116969] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2857.017802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2858.255537] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2862.008022] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2862.010059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.013341] Lustre: Skipped 5 previous similar messages [ 2862.015021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2862.018194] Lustre: Skipped 9 previous similar messages [ 2866.123489] Lustre: server umount lustre-OST0000 complete [ 2873.088578] Lustre: server umount lustre-MDT0000 complete [ 2873.975181] LustreError: 17924:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854509 with bad export cookie 10218445132457782015 [ 2873.976120] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2873.979973] LustreError: 17924:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2874.071776] Lustre: server umount lustre-MDT0001 complete [ 2876.687826] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2877.081337] Key type lgssc unregistered [ 2877.141714] LNet: 20840:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2878.146630] LNet: Removed LNI 192.168.202.133@tcp [ 2884.206008] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2884.292394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2884.296444] alg: No test for adler32 (adler32-zlib) [ 2885.095934] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 2885.140613] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2885.141996] LNet: Accept secure, port 988 [ 2886.665502] Key type lgssc registered [ 2886.851679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2888.548767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2888.554651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.669408] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2889.678686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.428241] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2892.461684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2892.551445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2893.219745] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2894.150817] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2894.652195] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2896.181077] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2896.184851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2896.274394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.404366] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2898.353067] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2898.591718] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 2899.101850] Lustre: server umount lustre-OST0000 complete [ 2899.982272] LustreError: 22435:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2899.985520] LustreError: 22435:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2899.989245] LustreError: 22435:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 2901.287949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.291421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.293143] Lustre: Skipped 1 previous similar message [ 2902.292006] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2902.293784] Lustre: Skipped 1 previous similar message [ 2906.048803] Lustre: server umount lustre-MDT0000 complete [ 2906.911051] LustreError: 22390:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854542 with bad export cookie 15707994080231041944 [ 2906.911194] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2906.918062] LustreError: 22390:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2907.002247] Lustre: server umount lustre-MDT0001 complete [ 2908.649768] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2909.051670] Key type lgssc unregistered [ 2909.113728] LNet: 26079:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2910.118733] LNet: Removed LNI 192.168.202.133@tcp [ 2914.627641] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2914.733309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2914.736568] alg: No test for adler32 (adler32-zlib) [ 2915.539879] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 2915.579543] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2915.580819] LNet: Accept secure, port 988 [ 2917.105473] Key type lgssc registered [ 2917.273716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2919.327981] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2919.330732] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2919.334016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2936.388525] LustreError: 13a-8: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2972.389499] LustreError: 27300:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 3002.405459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.491302] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3004.418998] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3004.943774] Lustre: server umount lustre-OST0000 complete [ 3007.884960] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3008.257674] Key type lgssc unregistered [ 3008.312659] LNet: 29016:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3009.320798] LNet: Removed LNI 192.168.202.133@tcp [ 3018.295787] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3018.395596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3018.410379] alg: No test for adler32 (adler32-zlib) [ 3019.212989] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 3019.261957] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3019.263838] LNet: Accept secure, port 988 [ 3020.789469] Key type lgssc registered [ 3020.953628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3024.132712] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3024.494541] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3024.803154] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3024.805222] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3025.089395] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3025.091663] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3025.315682] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3026.363696] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3027.645318] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3027.650655] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3028.658206] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3028.660237] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3029.619228] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3029.631265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3029.633869] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.643623] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3029.645978] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3029.648464] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3029.650936] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3029.653253] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3029.656723] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3029.659928] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3029.662946] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3029.665840] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3029.668514] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3029.671275] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3029.674202] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3029.676971] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3029.679953] Lustre: 684:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3031.855699] Lustre: server umount MGS complete [ 3032.169493] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.302937] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.317968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.329700] Lustre: lustre-MDT0000: Aborting client recovery [ 3032.605890] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3032.624532] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3032.634722] Lustre: 1261:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3032.637596] Lustre: 1261:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3032.640575] Lustre: 1261:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3032.643809] Lustre: 1261:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3032.646636] Lustre: 1261:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3032.649634] Lustre: 1261:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3032.652565] Lustre: 1261:0:(osd_scrub.c:1858:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3032.658062] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3032.973363] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 3032.975395] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3032.987920] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 3032.990207] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3032.998952] Lustre: 1355:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3033.001802] Lustre: 1355:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3033.004286] Lustre: 1355:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3033.006672] Lustre: 1355:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3033.009366] Lustre: 1355:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3033.012023] Lustre: 1355:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3033.014479] Lustre: 1355:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3033.084462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3033.086175] Lustre: Skipped 1 previous similar message [ 3033.090151] Lustre: lustre-OST0000: Aborting recovery [ 3033.322514] LDISKFS-fs (loop3): file extents enabled, maximum tree depth=5 [ 3033.324543] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3033.339967] LDISKFS-fs (loop3): file extents enabled, maximum tree depth=5 [ 3033.342165] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3033.350784] Lustre: 1451:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3033.353441] Lustre: 1451:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3033.356375] Lustre: 1451:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3033.359279] Lustre: 1451:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3033.362025] Lustre: 1451:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3033.364918] Lustre: 1451:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3033.367739] Lustre: 1451:0:(osd_scrub.c:1858:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3033.372563] Lustre: Found index 1 for lustre-OST0001, updating log [ 3033.373866] Lustre: Skipped 1 previous similar message [ 3042.391447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3042.391937] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3042.397163] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3042.400690] LustreError: 1192:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 3042.410649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3043.391948] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 3043.393413] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 3048.325866] LustreError: 1162:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 350us: rc = -2 [ 3062.759886] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3062.762866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.769520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3063.431948] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.432175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3063.440363] Lustre: Skipped 2 previous similar messages [ 3064.779158] Lustre: server umount lustre-MDT0000 complete [ 3065.017827] LustreError: 1147:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854700 with bad export cookie 3787076036350371743 [ 3065.018680] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3065.023405] LustreError: 1147:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3068.439979] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.440181] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.440450] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3068.440454] Lustre: Skipped 2 previous similar messages [ 3068.467373] LustreError: Skipped 1 previous similar message [ 3070.444045] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.444120] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3070.444122] Lustre: Skipped 1 previous similar message [ 3070.462043] LustreError: Skipped 1 previous similar message [ 3071.152570] Lustre: server umount lustre-MDT0001 complete [ 3077.495584] Lustre: server umount lustre-OST0000 complete [ 3083.925367] Lustre: server umount lustre-OST0001 complete [ 3086.169624] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 23:38:41 (1718854721) [ 3095.020801] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3097.399132] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3100.393589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3102.693231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3104.734716] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3104.740302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3106.575042] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3106.580806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3108.715312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3108.742474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.852757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3108.865973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3108.920739] Lustre: lustre-MDT0000: new disk, initializing [ 3108.951935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3108.955710] Lustre: Skipped 1 previous similar message [ 3108.963211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3110.092862] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3114.177525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3114.196169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3114.216822] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3114.227268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3114.229208] Lustre: Skipped 1 previous similar message [ 3114.266858] Lustre: lustre-MDT0001: new disk, initializing [ 3114.287535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3114.296276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3114.299599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3114.987044] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3117.665284] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3118.285163] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3120.344840] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3120.351251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3120.378060] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3120.383817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3120.495846] Lustre: lustre-OST0000: new disk, initializing [ 3120.499528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3120.532075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3122.333185] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3125.222659] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3125.251145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3125.256557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3125.286378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3127.750793] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3135.788476] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3140.541005] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3150.186833] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3195.384594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.384812] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3195.391329] Lustre: Skipped 1 previous similar message [ 3195.393101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3195.395411] Lustre: Skipped 2 previous similar messages [ 3197.494447] Lustre: server umount lustre-OST0000 complete [ 3199.416569] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3199.421734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.429503] Lustre: Skipped 1 previous similar message [ 3204.985392] Lustre: server umount lustre-MDT0000 complete [ 3205.399892] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3205.403977] LustreError: Skipped 1 previous similar message [ 3206.211807] LustreError: 6551:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854841 with bad export cookie 3787076036350374410 [ 3206.213627] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3206.225448] LustreError: 6551:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3206.373026] Lustre: server umount lustre-MDT0001 complete [ 3209.538138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.503993] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3212.284296] Lustre: server umount MGS complete [ 3216.306462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.484183] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.489548] LustreError: 13573:0:(qmt_pool.c:1521:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3216.508655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.681743] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3220.856562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3222.165596] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3223.711179] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.643179] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3226.936021] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3226.942181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3228.932336] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3230.434147] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3266.136628] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3266.136999] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.137002] Lustre: Skipped 1 previous similar message [ 3266.137781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3266.137783] Lustre: Skipped 3 previous similar messages [ 3266.158575] LustreError: Skipped 1 previous similar message [ 3268.725500] Lustre: server umount lustre-OST0000 complete [ 3271.064673] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3276.279681] Lustre: server umount lustre-MDT0000 complete [ 3277.654813] LustreError: 13543:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718854913 with bad export cookie 3787076036350376321 [ 3277.656546] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3277.667737] LustreError: 13543:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3277.820853] Lustre: server umount lustre-MDT0001 complete [ 3281.524414] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3282.333982] Key type lgssc unregistered [ 3282.425984] LNet: 17812:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3283.437046] LNet: Removed LNI 192.168.202.133@tcp [ 3295.258532] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3300.404417] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3300.597762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.602048] alg: No test for adler32 (adler32-zlib) [ 3301.427077] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 3301.496694] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3301.499934] LNet: Accept secure, port 988 [ 3303.048581] Key type lgssc registered [ 3303.378059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3306.381003] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3308.729400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3310.926755] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3310.935714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3313.213813] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3313.218833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3315.724380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3315.743083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3315.750801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.848250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3316.859530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3316.914920] Lustre: lustre-MDT0000: new disk, initializing [ 3316.948936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.959040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3318.070009] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3322.128409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3322.152435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3322.196581] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3322.219018] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3322.222370] Lustre: Skipped 1 previous similar message [ 3322.273841] Lustre: lustre-MDT0001: new disk, initializing [ 3322.310831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3322.328398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3322.334123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3323.467199] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3326.139819] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3326.618227] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3328.952873] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3328.960398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3328.989171] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3328.997123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3329.122869] Lustre: lustre-OST0000: new disk, initializing [ 3329.126292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3329.161949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.885007] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3332.229107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3332.234275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3332.268229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3333.439187] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3336.909799] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3340.889993] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3349.453268] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3397.352685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.353137] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3397.353661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3397.368123] Lustre: Skipped 1 previous similar message [ 3399.470251] Lustre: server umount lustre-OST0000 complete [ 3402.360541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.361104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.361106] Lustre: Skipped 1 previous similar message [ 3402.374979] Lustre: Skipped 1 previous similar message [ 3406.882676] Lustre: server umount lustre-MDT0000 complete [ 3407.367875] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.371710] LustreError: Skipped 1 previous similar message [ 3408.056716] LustreError: 22585:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718855043 with bad export cookie 7570852543624715481 [ 3408.058629] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3408.070373] LustreError: 22585:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3408.193845] Lustre: server umount lustre-MDT0001 complete [ 3411.185532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.359024] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3414.162619] Lustre: server umount MGS complete [ 3417.939401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3418.106723] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.111572] LustreError: 29587:0:(qmt_pool.c:1521:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3418.125649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.215719] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3421.891117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3422.995150] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3424.485051] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3425.366295] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3427.903452] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3427.908913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3428.054330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.058312] Lustre: Skipped 1 previous similar message [ 3429.917431] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3431.516260] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3447.080717] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3447.080771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.081210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3447.081212] Lustre: Skipped 1 previous similar message [ 3447.102696] LustreError: Skipped 1 previous similar message [ 3452.088429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3452.091840] Lustre: Skipped 2 previous similar messages [ 3452.818547] Lustre: server umount lustre-OST0000 complete [ 3457.064299] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3457.068851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.075749] Lustre: Skipped 1 previous similar message [ 3457.079069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3460.275259] Lustre: server umount lustre-MDT0000 complete [ 3461.678627] LustreError: 29601:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718855097 with bad export cookie 7570852543624717378 [ 3461.681209] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.691893] LustreError: 29601:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3461.809393] Lustre: server umount lustre-MDT0001 complete [ 3465.446785] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3466.221130] Key type lgssc unregistered [ 3466.309920] LNet: 1379:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3467.316943] LNet: Removed LNI 192.168.202.133@tcp [ 3476.332171] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3476.484988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3476.502774] alg: No test for adler32 (adler32-zlib) [ 3477.300815] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 3477.342642] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3477.344328] LNet: Accept secure, port 988 [ 3478.869537] Key type lgssc registered [ 3479.190160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3481.966071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3481.975110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.135370] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.147755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.209460] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3487.146225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3487.314125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3488.475202] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3490.037535] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3490.901045] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3493.383017] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3493.389644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3493.507892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3495.241829] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3496.815446] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3499.161737] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3499.165825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3499.184472] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3499.187004] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3499.220466] Lustre: lustre-OST0001: new disk, initializing [ 3499.222527] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3499.246529] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3501.002691] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3504.071144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3504.073359] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3504.080439] LustreError: 6604:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3504.082795] LustreError: 6604:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 3505.083107] LustreError: 6598:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3505.088686] LustreError: 6598:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 3506.088259] LustreError: 6604:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3508.097705] LustreError: 6604:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3508.103698] LustreError: 6604:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [ 3509.479722] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3509.519034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3509.999652] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012bfed800.ost_server_uuid 50 [ 3510.310773] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012bfed800.ost_server_uuid in FULL state after 0 sec [ 3511.052846] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012bfed800.ost_server_uuid 50 [ 3511.552120] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88012bfed800.ost_server_uuid in FULL state after 0 sec [ 3512.115196] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3523.556000] Lustre: lustre-OST0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 3527.128470] LustreError: 0-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3527.135661] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.144911] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3527.153187] Lustre: Skipped 1 previous similar message [ 3528.553261] Lustre: lustre-OST0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 3528.557419] Lustre: Skipped 3 previous similar messages [ 3529.145183] Lustre: server umount lustre-OST0001 complete [ 3532.568423] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3532.573241] LustreError: Skipped 1 previous similar message [ 3532.575974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.585552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.113061] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.120911] LustreError: Skipped 1 previous similar message [ 3536.789248] Lustre: server umount lustre-OST0000 complete [ 3538.120298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.128494] Lustre: Skipped 1 previous similar message [ 3538.131681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3538.135412] Lustre: Skipped 3 previous similar messages [ 3544.207062] Lustre: server umount lustre-MDT0000 complete [ 3545.607682] LustreError: 5931:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718855180 with bad export cookie 10533406826221514784 [ 3545.609695] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3545.627266] LustreError: 5931:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3545.762965] Lustre: server umount lustre-MDT0001 complete [ 3549.428872] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3550.188459] Key type lgssc unregistered [ 3550.277892] LNet: 8712:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3551.284878] LNet: Removed LNI 192.168.202.133@tcp [ 3559.357527] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3559.511468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3559.536138] alg: No test for adler32 (adler32-zlib) [ 3560.354180] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 3560.418427] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3560.421308] LNet: Accept secure, port 988 [ 3561.962542] Key type lgssc registered [ 3562.283402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3564.979585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3564.989277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.155500] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.174842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3567.251544] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3568.714619] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3571.117965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3571.265909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3572.352058] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3575.669378] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3575.675203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3575.814383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.813538] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.817432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:68 to 0x280000401:97) [ 3577.550923] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3579.026460] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3580.814798] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.823401] LustreError: Skipped 1 previous similar message [ 3581.445447] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3581.451235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3581.544232] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3583.341964] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3584.909946] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3586.462895] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3587.211975] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3587.292067] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3666.584220] LustreError: 0-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3666.588149] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.594334] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3669.695952] Lustre: server umount lustre-OST0001 complete [ 3671.688738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.689770] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3671.690012] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3671.690014] Lustre: Skipped 1 previous similar message [ 3671.709320] Lustre: Skipped 2 previous similar messages [ 3676.696522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.696532] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.696534] LustreError: Skipped 1 previous similar message [ 3676.711393] Lustre: Skipped 2 previous similar messages [ 3677.201029] Lustre: server umount lustre-OST0000 complete [ 3681.432523] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3681.437280] LustreError: Skipped 1 previous similar message [ 3681.439928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.447954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3684.706452] Lustre: server umount lustre-MDT0000 complete [ 3686.037101] LustreError: 10211:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718855321 with bad export cookie 5687096211005324379 [ 3686.038100] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3686.044180] LustreError: 10211:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3686.198912] Lustre: server umount lustre-MDT0001 complete [ 3689.889135] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3690.673256] Key type lgssc unregistered [ 3690.766548] LNet: 15549:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3691.773981] LNet: Removed LNI 192.168.202.133@tcp [ 3708.728573] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3708.879958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3708.884214] alg: No test for adler32 (adler32-zlib) [ 3709.704450] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 3709.762384] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3709.765176] LNet: Accept secure, port 988 [ 3711.307802] Key type lgssc registered [ 3711.675459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3714.993618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3715.006491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.184939] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.201726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.281990] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3720.483588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3720.646561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3721.758381] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3722.826447] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 3725.179087] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3725.184843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3725.323544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.323597] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3726.326097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:68 to 0x280000401:129) [ 3727.090838] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3730.148253] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3730.154140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3730.258606] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3731.260411] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 3732.065656] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3736.255158] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3741.992204] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 3750.664464] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3750.669414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.677938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3751.289307] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.290036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3751.303396] Lustre: Skipped 2 previous similar messages [ 3756.296249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3756.299709] Lustre: Skipped 5 previous similar messages [ 3756.674803] Lustre: server umount lustre-MDT0000 complete [ 3758.008765] LustreError: 18195:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718855393 with bad export cookie 12597143444988991881 [ 3758.010711] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3758.023802] LustreError: 18195:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3758.159229] Lustre: server umount lustre-MDT0001 complete [ 3769.330115] Lustre: server umount lustre-OST0000 complete [ 3780.766525] Lustre: server umount lustre-OST0001 complete [ 3785.617375] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3788.743544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.940976] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.960763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.075119] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3792.744375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3793.538221] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3794.224198] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 3795.785132] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3795.787843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3796.976657] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3798.874744] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3798.877910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:68 to 0x280000401:161) [ 3799.140936] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3799.146897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3799.249302] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3799.254088] Lustre: Skipped 2 previous similar messages [ 3800.817659] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 3801.003990] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3807.693586] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3808.930469] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3817.848830] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3817.849878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3817.859062] Lustre: Skipped 2 previous similar messages [ 3820.918556] Lustre: server umount lustre-MDT0000 complete [ 3822.263823] LustreError: 28381:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718855457 with bad export cookie 12597143444988993540 [ 3822.266433] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3822.276825] LustreError: 28381:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3822.414108] Lustre: server umount lustre-MDT0001 complete [ 3833.881371] Lustre: server umount lustre-OST0000 complete [ 3845.311576] Lustre: server umount lustre-OST0001 complete [ 3846.617645] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 3847.188075] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 3847.782091] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 23:51:22 (1718855482) [ 3854.525820] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3857.119800] SGI XFS with ACLs, security attributes, no debug enabled [ 3857.126082] XFS (loop0): Mounting V5 Filesystem [ 3857.138522] XFS (loop0): Ending clean mount [ 3859.410325] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [ 3859.531691] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3865.371577] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3866.675449] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3869.218048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3871.518535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3873.660408] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3873.663513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3875.840284] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3875.846061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3878.333660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3878.356704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.455614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3878.468078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3878.521426] Lustre: lustre-MDT0000: new disk, initializing [ 3878.551222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.561333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3879.706458] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3883.916288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3883.942461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3884.000444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3884.003689] Lustre: Skipped 1 previous similar message [ 3884.053906] Lustre: lustre-MDT0001: new disk, initializing [ 3884.099389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3884.105074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3885.108057] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3887.473036] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3887.990763] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3889.542310] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3889.544938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3889.559266] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3889.561831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3889.628161] Lustre: lustre-OST0000: new disk, initializing [ 3889.630793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3890.766545] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3891.552451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3891.559289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3891.587335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3893.390200] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3894.599338] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3894.680764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3895.830389] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3895.907943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3896.583955] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3896.584531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.584532] Lustre: Skipped 1 previous similar message [ 3896.584779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3896.584780] Lustre: Skipped 3 previous similar messages [ 3896.592339] LustreError: Skipped 1 previous similar message [ 3902.609418] Lustre: server umount lustre-OST0000 complete [ 3904.104458] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3904.108986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.115730] Lustre: Skipped 1 previous similar message [ 3906.600334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3906.603811] Lustre: Skipped 4 previous similar messages [ 3910.068608] Lustre: server umount lustre-MDT0000 complete [ 3911.447665] LustreError: 3072:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718855546 with bad export cookie 12597143444988995094 [ 3911.449898] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3911.459718] LustreError: 3072:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3911.578085] Lustre: server umount lustre-MDT0001 complete [ 3914.370468] XFS (loop0): Unmounting Filesystem [ 3915.924000] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 23:52:30 (1718855550) [ 3918.406375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.585534] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.605850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.607994] Lustre: Skipped 2 previous similar messages [ 3919.730095] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3922.788121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3924.058194] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3925.573608] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3926.439037] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3928.829599] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3928.836824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3930.694300] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3932.241918] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3953.000533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.000725] LustreError: 0-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3953.012001] Lustre: Skipped 1 previous similar message [ 3953.014944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3953.018664] Lustre: Skipped 1 previous similar message [ 3956.382503] Lustre: server umount lustre-OST0000 complete [ 3957.976398] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3963.772132] Lustre: server umount lustre-MDT0000 complete [ 3965.166851] LustreError: 7045:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718855600 with bad export cookie 12597143444988995948 [ 3965.169304] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.178930] LustreError: 7045:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3965.309652] Lustre: server umount lustre-MDT0001 complete [ 3969.017631] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3969.762290] Key type lgssc unregistered [ 3969.849167] LNet: 10833:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3970.856933] LNet: Removed LNI 192.168.202.133@tcp [ 3979.599000] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3979.727316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3979.751177] alg: No test for adler32 (adler32-zlib) [ 3980.549633] Lustre: Lustre: Build Version: 2.15.63_196_g38fa724 [ 3980.589671] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3980.591014] LNet: Accept secure, port 988 [ 3982.113459] Key type lgssc registered [ 3982.333170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3984.408705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3984.417736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.595453] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3985.606288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.736544] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3989.825048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3989.968210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3991.086964] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3992.616514] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3993.505856] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3995.934068] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3995.941349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3996.094184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3997.806852] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3999.270086] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4019.461286] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 4020.008602] LustreError: 0-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4020.015540] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.028626] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4022.951555] Lustre: server umount lustre-MDT0001 complete [ 4023.997617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4024.457338] LustreError: 137-5: lustre-MDT0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4026.573558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4026.605705] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4026.607146] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4026.616573] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4026.689168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4027.764462] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4031.707075] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 4032.056890] LustreError: 16492:0:(obd_mount.c:209:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4032.064778] LustreError: 16492:0:(tgt_mount.c:681:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4032.072600] LustreError: 12534:0:(mgc_request.c:609:do_requeue()) failed processing log: -17 [ 4032.099245] LustreError: 16497:0:(obd_config.c:1991:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -17 [ 4032.106693] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 4329.956688] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 4329.959792] Lustre: Skipped 1 previous similar message [ 4332.185004] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.191247] Lustre: Skipped 1 previous similar message [ 4332.193878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4332.196510] Lustre: Skipped 1 previous similar message [ 4335.809323] Lustre: server umount lustre-MDT0001 complete [ 4336.861077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4337.192171] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4337.195649] LustreError: Skipped 3 previous similar messages [ 4339.230564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4339.271724] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4339.276553] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4339.296006] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4339.401072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4340.518750] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4344.410760] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 4345.721891] LustreError: 1028:0:(obd_mount.c:209:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4345.726600] LustreError: 1028:0:(tgt_mount.c:681:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4345.733243] LustreError: 12534:0:(mgc_request.c:609:do_requeue()) failed processing log: -17 [ 4345.737590] LustreError: 12534:0:(mgc_request.c:609:do_requeue()) Skipped 1 previous similar message [ 4345.761546] LustreError: 1033:0:(obd_config.c:700:class_setup()) Device 8 already setup (type osp) [ 4345.766401] LustreError: 1033:0:(obd_config.c:1991:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -17 [ 4345.772178] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.133@tcp [ 4644.888712] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4644.894137] Lustre: Skipped 1 previous similar message [ 4644.896627] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4644.898596] Lustre: Skipped 2 previous similar messages [ 4648.949937] Lustre: server umount lustre-MDT0001 complete [ 4649.699489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4649.896000] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.900131] LustreError: Skipped 1 previous similar message [ 4651.710156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4651.738178] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4651.741229] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4651.754564] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4651.844824] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4652.698051] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4656.857375] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 4659.828190] LustreError: 18414:0:(obd_mount.c:209:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4659.832174] LustreError: 18414:0:(tgt_mount.c:681:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4659.836246] LustreError: 12534:0:(mgc_request.c:609:do_requeue()) failed processing log: -17 [ 4659.839173] LustreError: 12534:0:(mgc_request.c:609:do_requeue()) Skipped 1 previous similar message [ 4659.850779] LustreError: 18425:0:(lod_lov.c:231:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 4659.855112] LustreError: 18425:0:(obd_config.c:1991:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -114 [ 4659.858769] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 4967.319929] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4967.322720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4967.326521] Lustre: Skipped 1 previous similar message [ 4967.328619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4967.330978] Lustre: Skipped 1 previous similar message [ 4972.011550] Lustre: server umount lustre-MDT0000 complete [ 4972.360402] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4972.365380] LustreError: Skipped 3 previous similar messages [ 4973.152540] LustreError: 12522:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718856608 with bad export cookie 17660432105538492890 [ 4973.153589] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4973.161925] LustreError: 12522:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4977.369453] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.374222] Lustre: Skipped 2 previous similar messages [ 4979.278615] Lustre: server umount lustre-MDT0001 complete [ 4980.350836] Lustre: server umount lustre-OST0000 complete [ 4982.392804] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 4983.985469] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4986.276595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4987.962954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4989.456528] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4989.463737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4991.064236] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4991.068368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4992.556005] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 00:10:27 (1718856627) [ 4994.151202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4994.168166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4994.233365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4994.243851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4994.277446] Lustre: lustre-MDT0000: new disk, initializing [ 4994.295751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4994.301973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4995.084384] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4998.434735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4998.450280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4998.471330] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4998.482832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4998.484440] Lustre: Skipped 1 previous similar message [ 4998.516260] Lustre: lustre-MDT0001: new disk, initializing [ 4998.537107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4998.546979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4998.550599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4999.212141] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5001.352975] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5001.858655] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5003.428632] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5003.432731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5003.449261] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5003.452360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5003.518597] Lustre: lustre-OST0000: new disk, initializing [ 5003.520314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5003.534984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5004.655823] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5006.859580] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5011.683249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5011.687467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5011.709892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5016.696046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5016.696538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5016.696539] Lustre: Skipped 4 previous similar messages [ 5016.702624] Lustre: Skipped 2 previous similar messages [ 5020.058654] Lustre: server umount lustre-MDT0000 complete [ 5020.959191] LustreError: 5767:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718856656 with bad export cookie 17660432105538494787 [ 5020.959919] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5020.960336] LustreError: 6736:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 5020.968959] LustreError: 5767:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5021.055445] Lustre: server umount lustre-MDT0001 complete [ 5032.058120] Lustre: server umount lustre-OST0000 complete [ 5035.080496] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:11:10 (1718856670) [ 5039.550810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5039.688893] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5039.695479] LustreError: Skipped 2 previous similar messages [ 5039.709045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.566392] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5043.280897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5044.411098] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5046.281031] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5047.357592] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5048.514535] Lustre: Failing over lustre-MDT0000 [ 5048.606949] Lustre: server umount lustre-MDT0000 complete [ 5053.420456] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5053.425002] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.464187] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.476534] LustreError: Skipped 5 previous similar messages [ 5063.259890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5063.335535] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5063.344278] Lustre: 11137:0:(mgc_request.c:1838:mgc_process_log()) MGC192.168.202.133@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 5063.549037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5063.555201] Lustre: Skipped 1 previous similar message [ 5063.592042] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5065.290079] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5068.535961] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5068.538493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 5068.552023] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5068.634221] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 5073.560888] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5073.574683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5073.580319] Lustre: Skipped 2 previous similar messages [ 5076.521226] Lustre: server umount lustre-MDT0000 complete [ 5077.978487] LustreError: 13800:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718856713 with bad export cookie 17660432105538496642 [ 5077.980092] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5077.988609] LustreError: 13800:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5078.153347] Lustre: server umount lustre-MDT0001 complete [ 5081.344522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.516884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5082.446798] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5085.340031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5086.347232] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5087.673303] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5088.458402] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5089.134768] Lustre: Failing over lustre-MDT0000 [ 5089.137109] LustreError: 14718:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5089.142429] LustreError: 14718:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5089.146241] LustreError: 14718:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 5089.212761] Lustre: server umount lustre-MDT0000 complete [ 5090.456490] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5090.459962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.465462] Lustre: Skipped 3 previous similar messages [ 5095.491625] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.496689] LustreError: Skipped 4 previous similar messages [ 5102.048809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5102.104523] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5102.235562] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5103.248903] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5106.282804] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 5107.209423] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5107.210722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 5107.210724] Lustre: Skipped 1 previous similar message [ 5107.227157] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5114.397999] Lustre: server umount lustre-MDT0000 complete [ 5115.644398] LustreError: 14670:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718856750 with bad export cookie 17660432105538497524 [ 5115.646821] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5115.654250] LustreError: 14670:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5115.792263] Lustre: server umount lustre-MDT0001 complete [ 5118.809711] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:12:33 (1718856753) [ 5124.270957] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 5126.473331] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5129.754921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5131.894954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5133.724897] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5133.732654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5135.564668] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5135.571584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5137.652198] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5140.674581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5140.693651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.765856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5140.775990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5140.811044] Lustre: lustre-MDT0000: new disk, initializing [ 5140.842120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5140.845388] Lustre: Skipped 2 previous similar messages [ 5140.851823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5141.723719] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5146.032126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5146.059261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5146.117906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5146.121409] Lustre: Skipped 1 previous similar message [ 5146.159976] Lustre: lustre-MDT0001: new disk, initializing [ 5146.224826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5146.230309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5147.397299] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5150.234040] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5151.281236] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5153.659200] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5153.664480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5153.688104] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5153.693815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5155.366382] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5157.467230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5157.473450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5157.492436] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 5157.502152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5157.507904] LustreError: 23791:0:(ofd_dev.c:1597:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x280000401:33, last_id 65281. Likely MDS last_id corruption [ 5157.514264] LustreError: 24411:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -22 [ 5157.517914] LustreError: 24411:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -22 [ 5158.145822] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5160.941589] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 5167.498994] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5167.502166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5167.508457] Lustre: Skipped 3 previous similar messages [ 5167.511277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5167.526843] Lustre: Skipped 5 previous similar messages [ 5173.218834] Lustre: server umount lustre-OST0000 complete [ 5175.052841] LustreError: 24411:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5175.058800] LustreError: 24411:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5176.233874] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5176.237317] LustreError: Skipped 2 previous similar messages [ 5181.170165] Lustre: server umount lustre-MDT0000 complete [ 5182.556185] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5182.565673] LustreError: Skipped 5 previous similar messages [ 5183.800954] LustreError: 22736:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718856819 with bad export cookie 17660432105538497923 [ 5183.802809] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5183.831531] LustreError: 22736:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5184.222084] Lustre: server umount lustre-MDT0001 complete [ 5187.704567] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5189.653837] Lustre: DEBUG MARKER: conf-sanity test_122a: @@@@@@ FAIL: test_122a failed with 203 [ 5196.516001] Lustre: DEBUG MARKER: == conf-sanity test 123aa: llog_print works with FIDs and simple names ========================================================== 00:13:51 (1718856831) [ 5200.045516] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5203.832435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5203.845233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5205.886002] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5210.380069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5210.595966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5210.599007] Lustre: Skipped 3 previous similar messages [ 5212.011823] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5214.013781] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5215.436422] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5219.004563] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5219.015925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5220.293805] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 5221.755296] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5224.094673] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5236.231083] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:14:30 (1718856870) [ 5237.343963] Lustre: Modifying parameter general.jobid_name in log params [ 5237.345704] Lustre: Skipped 1 previous similar message [ 5242.083535] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:14:36 (1718856876) [ 5246.711824] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:14:41 (1718856881) [ 5252.211320] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:14:46 (1718856886) [ 5265.306900] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:15:00 (1718856900) [ 5266.977294] Lustre: *** cfs_fail_loc=131b, val=2*** [ 5268.939993] Lustre: *** cfs_fail_loc=131b, val=2*** [ 5273.353174] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:15:08 (1718856908) [ 5274.275318] Lustre: Modifying parameter general.jobid_name in log params [ 5274.278129] Lustre: Skipped 16 previous similar messages [ 5280.948675] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:15:15 (1718856915) [ 5292.488968] Lustre: server umount lustre-OST0000 complete [ 5295.688539] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5295.696132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.702204] Lustre: Skipped 5 previous similar messages [ 5295.705327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.711453] Lustre: Skipped 7 previous similar messages [ 5300.684591] Lustre: server umount lustre-MDT0000 complete [ 5302.595330] LustreError: 30063:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718856937 with bad export cookie 17660432105538534015 [ 5302.601622] LustreError: 30063:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5302.602177] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5302.778648] Lustre: server umount lustre-MDT0001 complete [ 5305.090389] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5312.343915] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 5315.498779] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5319.269555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5321.927841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5324.385604] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5324.395026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5326.804320] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5326.812999] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5329.632162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5329.655878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5329.665622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.797870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5329.800597] Lustre: Skipped 1 previous similar message [ 5329.848795] Lustre: lustre-MDT0000: new disk, initializing [ 5329.850776] Lustre: Skipped 1 previous similar message [ 5329.898119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5331.120508] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5336.275672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5336.296391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5336.477461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5337.674524] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5340.851597] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5342.007885] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5344.669761] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5344.674987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5344.707549] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5344.716984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5344.878097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5344.881075] Lustre: Skipped 3 previous similar messages [ 5345.246875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5345.318064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5346.896919] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5350.183938] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5351.528933] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5351.632255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5352.934247] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5353.002683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5355.289065] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5359.891280] Lustre: server umount lustre-OST0000 complete [ 5367.396528] Lustre: server umount lustre-MDT0000 complete [ 5369.106843] LustreError: 7135:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718857004 with bad export cookie 17660432105538536206 [ 5369.109985] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5369.120962] LustreError: 7135:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5369.271732] Lustre: server umount lustre-MDT0001 complete [ 5373.816136] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:16:48 (1718857008) [ 5376.900906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.150765] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.165995] LustreError: Skipped 2 previous similar messages [ 5378.555900] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5382.100277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5383.475214] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5385.007168] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5385.857079] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5388.397167] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5388.405027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5390.272371] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5391.744099] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5408.746190] LustreError: 14454:0:(llog.c:436:llog_init_handle()) llog type is not specified! [ 5409.222617] Lustre: Modifying parameter general.timeout in log params [ 5409.224263] Lustre: Skipped 6 previous similar messages [ 5461.167185] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:18:16 (1718857096) [ 5467.400187] LustreError: 0-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5467.403923] LustreError: Skipped 1 previous similar message [ 5472.113778] Lustre: server umount lustre-MDT0000 complete [ 5473.514840] LustreError: 11936:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718857108 with bad export cookie 17660432105538536990 [ 5473.521562] LustreError: 11936:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5473.672857] Lustre: server umount lustre-MDT0001 complete [ 5485.197685] Lustre: server umount lustre-OST0000 complete [ 5488.464654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5489.856371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5491.457703] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5491.465568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5493.107931] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5493.117495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5495.783227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.923593] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5497.207099] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5500.744112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5500.797941] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5502.175029] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5503.902860] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5504.810415] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5507.286958] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5507.295868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5507.372412] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5509.463721] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5510.892127] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5519.096436] Lustre: server umount lustre-OST0000 complete [ 5526.734815] Lustre: server umount lustre-MDT0000 complete [ 5527.958646] LustreError: 0-0: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5527.966305] LustreError: Skipped 1 previous similar message [ 5528.115432] Lustre: server umount lustre-MDT0001 complete [ 5529.714906] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5530.461549] Lustre: DEBUG MARKER: conf-sanity test_123F: @@@@@@ FAIL: test_123F failed with 203 [ 5533.159165] Lustre: DEBUG MARKER: == conf-sanity test 123G: clear and reset all parameters using apply_yaml ========================================================== 00:19:28 (1718857168) [ 5535.907014] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5538.977198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5538.986666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.575237] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5544.388931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5545.882270] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5547.411186] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5548.172023] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5550.500588] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5550.506771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5552.348822] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5554.064657] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5567.215060] Lustre: Modifying parameter general.jobid_var in log params [ 5567.218015] Lustre: Skipped 136 previous similar messages [ 5570.079935] Lustre: server umount lustre-MDT0000 complete [ 5571.564333] LustreError: 31647:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1718857206 with bad export cookie 17660432105538540042 [ 5571.570302] LustreError: 31647:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5573.720609] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.723127] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5573.723129] Lustre: Skipped 17 previous similar messages [ 5573.737425] Lustre: Skipped 13 previous similar messages [ 5577.757567] Lustre: server umount lustre-MDT0001 complete [ 5593.175607] Lustre: server umount lustre-OST0000 complete [ 5596.991045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5598.782391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5600.486761] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5600.495194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5602.037963] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5602.047122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5605.300630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5605.473603] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5605.598091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5605.601342] Lustre: Skipped 9 previous similar messages [ 5607.022584] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5610.828934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5610.877134] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5612.366705] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5614.186302] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5615.234921] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5618.261101] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5618.273767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5618.386571] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5620.501258] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5622.450050] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5638.296082] LustreError: 0-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5638.300072] LustreError: Skipped 3 previous similar messages [ 5641.331375] Lustre: server umount lustre-OST0000 complete [ 5648.626585] Lustre: server umount lustre-MDT0000 complete [ 5649.913658] Lustre: server umount lustre-MDT0001 complete [ 5651.508380] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5652.356339] Lustre: DEBUG MARKER: conf-sanity test_123G: @@@@@@ FAIL: test_123G failed with 203