This patch improves the performance of LDM and STM instruction
emulation. This is desirable because.
- jprobes and kretprobes probe the first instruction in a function and,
when the frame pointer is omitted, this instruction is often a STM
used to push registers onto the stack.
- The STM and LDM instructions are common in the body and tail of
functions.
- At the same time as being a common instruction form, they also have
one of the slowest and most complicated simulation routines.
The approach taken to optimisation is to use emulation rather than
simulation, that is, a modified form of the instruction is run with
an appropriate register context.
Benchmarking on an OMAP3530 shows the optimised emulation is between 2
and 3 times faster than the simulation routines. On a Kirkwood based
device the relative performance was very significantly better than this.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
The encoding of these instructions is substantially the same for both
ARM and Thumb, so we can have common decoding and simulation functions.
This patch moves the simulation functions from kprobes-arm.c to
kprobes-common.c. It also adds a new simulation function
(simulate_ldm1_pc) for the case where we load into PC because this may
need to interwork.
The instruction decoding is done by a custom function
(kprobe_decode_ldmstm) rather than just relying on decoding table
entries because we will later be adding optimisation code.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
This writes a value to PC which was obtained as the result of a
LDR or LDM instruction. For ARMv5T and later this must perform
interworking.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
For hints which may have observable effects, like SEV (send event), we
use kprobe_emulate_none which emulates the hint by executing the
original instruction.
For NOP we simulate the instruction using kprobe_simulate_nop, which
does nothing. As probes execute with interrupts disabled this is also
used for hints which may block for an indefinite time, like WFE (wait
for event).
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
These are very rare and/or problematic to emulate so we will take the
easy option and disallow probing them (as does the existing ARM
implementation).
Rejecting these instructions doesn't actually require any entries in the
decoding table as it is the default case for instructions which aren't
found.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
We previously changed the behaviour of probes so that conditional
instructions don't fire when the condition isn't met. For ARM branches,
and Thumb branches in IT blocks, this means they don't fire if the
branch isn't taken.
For consistency, we implement the same for Thumb conditional branch
instructions. This involves setting up insn_check_cc to point to the
relevant condition checking function. As the emulation routine is only
called when this condition passes, it doesn't need to check again and
can unconditionally update PC.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
SVC (SWI) instructions shouldn't occur in kernel code so we don't
need to be able to probe them.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
The normal Thumb singlestepping routine updates the IT state after
calling the instruction handler. We don't what this to happen after the
IT instruction simulation sets the IT state, therefore we need to
provide a custom singlestep routine.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
These instructions are equivalent to
stmdb sp!,{r0-r7,lr}
ldmdb sp!,{r0-r7,pc}
and we emulate them by transforming them into the 32-bit Thumb
instructions
stmdb r9!,{r0-r7,r8}
ldmdb r9!,{r0-r7,r8}
This is simpler, and almost certainly executes faster, than writing
simulation functions.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Most of these instructions only operate on the low registers R0-R7
so they can make use of t16_emulate_loregs_rwflags.
The instructions which use SP or PC for addressing have their own
simulation functions.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
These data-processing instructions operate on the full range of CPU
registers, so to simulate them we have to modify the registers used
by the instruction. We can't make use of the decoding table framework to
do this because the registers aren't encoded cleanly in separate
nibbles, therefore we need a custom decode function.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
This writes a value to PC, with interworking. I.e. switches to Thumb or
ARM mode depending on the state of the least significant bit.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
These instructions only operate on the low registers R0-R7, therefore
it is possible to emulate them by executing the original instruction
unaltered if we restore and save these registers. This is what
t16_emulate_loregs does.
Some of these instructions don't update the PSR when they execute in an
IT block, so there are two flavours of emulation functions:
t16_emulate_loregs_{noit}rwflags
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
For hints which may have observable effects, like SEV (send event), we
use kprobe_emulate_none which emulates the hint by executing the
original instruction.
For NOP we simulate the instruction using kprobe_simulate_nop, which
does nothing. As probes execute with interrupts disabled this is also
used for hints which may block for an indefinite time, like WFE (wait
for event).
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
The existing ARM instruction decoding functions are a mass of if/else
code. Rather than follow this pattern for Thumb instruction decoding
this patch implements an infrastructure for a new table driven scheme.
This has several advantages:
- Reduces the kernel size by approx 2kB. (The ARM instruction decoding
will eventually have -3.1kB code, +1.3kB data; with similar or better
estimated savings for Thumb decoding.)
- Allows programmatic checking of decoding consistency and test case
coverage.
- Provides more uniform source code and is therefore, arguably, clearer.
For a detailed explanation of how decoding tables work see the in-source
documentation in kprobes.h, and also for kprobe_decode_insn().
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
When we come to emulating Thumb instructions then, to interwork
correctly, the code on in the instruction slot must be invoked with a
function pointer which has the least significant bit set. Rather that
set this by hand in every Thumb emulation function we will add a new
field for this purpose to arch_specific_insn, called insn_fn.
This also enables us to seamlessly share emulation functions between ARM
and Thumb code.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
When a probe fires we must single-step the instruction which was
replaced by a breakpoint. As the steps to do this vary between ARM and
Thumb instructions we need a way to customise single-stepping.
This is done by adding a new hook called insn_singlestep to
arch_specific_insn which is initialised by the instruction decoding
functions.
These single-step hooks must update PC and call the instruction handler.
For Thumb instructions an additional step of updating ITSTATE is needed.
We do this after calling the handler because some handlers will need to
test if they are running in an IT block.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Now we no longer trigger probes on conditional instructions when the
condition is false, we can make use of conditional instructions as
breakpoints in ARM code to avoid taking unnecessary exceptions.
Note, we can't rely on not getting an exception when the condition check
fails, as that is Implementation Defined on newer ARM architectures. We
therefore still need to perform manual condition checks as well.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
This patch changes the behavior of kprobes on ARM so that:
Kprobes on conditional instructions don't trigger when the
condition is false. For conditional branches, this means that
they don't trigger in the branch not taken case.
Rationale:
When probes are placed onto conditionally executed instructions in a
Thumb IT block, they may not fire if the condition is not met. This
is because we use invalid instructions for breakpoints and "it is
IMPLEMENTATION DEFINED whether the instruction executes as a NOP or
causes an Undefined Instruction exception". Therefore, for consistency,
we will ignore all probes on any conditional instructions when the
condition is false. Alternative solutions seem to be too complex to
implement or inconsistent.
This issue was discussed on linux.arm.kernel in the thread titled
"[RFC] kprobes with thumb2 conditional code" See
http://comments.gmane.org/gmane.linux.linaro.devel/2985
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
This advances the ITSTATE bits in CPSR to their values for the next
instruction.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Extend the breakpoint insertion and catching functions to support Thumb
code.
As breakpoints are no longer of a fixed size, the flush_insns macro
is modified to take a size argument instead of an instruction count.
Note, we need both 16- and 32-bit Thumb breakpoints, because if we
were to use a 16-bit breakpoint to replace a 32-bit instruction which
was in an IT block, and the condition check failed, then the breakpoint
may not fire (it's unpredictable behaviour) and the CPU could then try
and execute the second half of the 32-bit Thumb instruction.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Extend arch_prepare_kprobe to support probing of Thumb code. For
the actual decoding of Thumb instructions, stub functions are
added which currently just reject the probe.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Fix up kprobes framework so that it builds and correctly interworks on
Thumb-2 kernels.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
The str_pc_offset value is architecturally defined on ARMv7 onwards so
we can make it a compile time constant. This means on Thumb kernels the
runtime checking code isn't needed, which saves us from having to fix it
to work for Thumb.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Move str_pc_offset into kprobes-common.c as it will be needed by common
code later.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
This file will contain the instruction decoding and emulation code
which is common to both ARM and Thumb instruction sets.
For now, we will just move over condition_checks from kprobes-arm.c
This table is also renamed to kprobe_condition_checks to avoid polluting
the public namespace with a too generic name.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Later, we will be adding a considerable amount of internal
implementation definitions to kprobe header files and it would be good
to have these in local header file along side the source code, rather
than pollute the existing header which is include by all users of
kprobes.
To this end, we add arch/arm/kernel/kprobes.h and move into this the
existing internal defintions from arch/arm/include/asm/kprobes.h
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
This file contains decoding and emulation functions for the ARM
instruction set. As we will later be adding a file for Thumb and a
file with common decoding functions, this renaming makes things clearer.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
This patch allows undef_hook's to be specified for 32-bit Thumb
instructions and also to be used for thumb kernel-side code.
32-bit Thumb instructions are specified in the form:
((first_half << 16 ) | second_half)
which matches the layout used by the ARM ARM.
ptrace was handling 32-bit Thumb instructions by hooking the first
halfword and manually checking the second half. This method would be
broken by this patch so it is migrated to make use of the new Thumb-2
support.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
The implementation of svc_exit didn't take into account any stack hole
created by svc_entry; as happens with the undef handler when kprobes are
configured. The fix is to read the saved value of SP rather than trying
to calculate it.
Signed-off-by: Jon Medhurst <tixy@yxit.co.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
To get hundredths of MHz the rate needs to be divided by 10'000.
Here is an example:
twd_timer_rate = 123456789
Before the patch:
twd_timer_rate / 1000000 = 123
(twd_timer_rate / 1000000) % 100 = 23
Result: 123.23MHz.
After being fixed:
twd_timer_rate / 1000000 = 123
(twd_timer_rate / 10000) % 100 = 45
Result: 123.45MHz.
Signed-off-by: Vitaly Kuzmichev <vkuzmichev@mvista.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Ensure that the meminfo array is sanity checked before we pass the
memory to memblock. This helps to ensure that memblock and meminfo
agree on the dimensions of memory, especially when more memory is
passed than the kernel can deal with.
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
armpmu_enable can be called in situations where no events are present
(for example, from the event rotation tick after a profiled task has
exited). In this case, we currently start the PMU anyway which may
leave it active inevitably without any events being monitored.
This patch adds a simple check to the enabling code so that we avoid
starting the PMU when no events are present.
Cc: <stable@kernel.org>
Reported-by: Ashwin Chaugle <ashwinc@codeaurora.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
When we bring a CPU online, we should wait for it to become active
before entering the idle thread, so we know that the scheduler and
thread migration is going to work.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
The "Thumb bit" of a symbol is only really meaningful for function
symbols (STT_FUNC).
However, sometimes a branch is relocated against a non-function
symbol; for example, PC-relative branches to anonymous assembler
local symbols are typically fixed up against the start-of-section
symbol, which is not a function symbol. Some inline assembler
generates references of this type, such as fixup code generated by
macros in <asm/uaccess.h>.
The existing relocation code for R_ARM_THM_CALL/R_ARM_THM_JUMP24
interprets this case as an error, because the target symbol appears
to be an ARM symbol; but this is really not the case, since the
target symbol is just a base in these cases. The addend defines
the precise offset to the target location, but since the addend is
encoded in a non-interworking Thumb branch instruction, there is no
explicit Thumb bit in the addend. Because these instructions never
interwork, the implied Thumb bit in the addend is 1, and the
destination is Thumb by definition.
This patch removes the extraneous Thumb bit check for non-function
symbols, enabling modules containing the affected relocation types
to be loaded. No modification to the actual relocation code is
required, since this code does not take bit[0] of the
location->destination offset into account in any case.
Function symbols are always checked for interworking conflicts, as
before.
Signed-off-by: Dave Martin <dave.martin@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Dump out the following 16-bit instruction to the faulting instruction
in the Code: line. This allows Thumb-2 instructions to be properly
encoded.
Tested-by: Kevin Hilman <khilman@ti.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
If the page to cmpxchg is user mode read only (not write),
we should simulate a data abort first.
Signed-off-by: Po-Yu Chuang <ratbert@faraday-tech.com>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
If the DT physical address is zero, this is equivalent to no DT.
Especially when the actual RAM physical address is not located at zero,
the result of phys_to_virt() would point to la-la-land and crash the
kernel, which crash is completely silent this early during boot.
Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Acked-by: Grant Likely <grant.likely@secretlab.ca>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
This patch fixes the lockdep warning of "unannotated irqs-off"[1].
After entering __irq_usr, arm core will disable interrupt automatically,
but __irq_usr does not annotate the irq disable, so lockdep may complain
the warning if it has chance to check this in irq handler.
This patch adds trace_hardirqs_off in __irq_usr before entering irq_handler
to handle the irq, also calls ret_to_user_from_irq to avoid calling
disable_irq again.
This is also a fix for irq off tracer.
[1], lockdep warning log of "unannotated irqs-off"
[ 13.804687] ------------[ cut here ]------------
[ 13.809570] WARNING: at kernel/lockdep.c:3335 check_flags+0x78/0x1d0()
[ 13.816467] Modules linked in:
[ 13.819732] Backtrace:
[ 13.822357] [<c01cb42c>] (dump_backtrace+0x0/0x100) from [<c06abb14>] (dump_stack+0x20/0x24)
[ 13.831268] r6:c07d8c2c r5:00000d07 r4:00000000 r3:00000000
[ 13.837280] [<c06abaf4>] (dump_stack+0x0/0x24) from [<c01ffc04>] (warn_slowpath_common+0x5c/0x74)
[ 13.846649] [<c01ffba8>] (warn_slowpath_common+0x0/0x74) from [<c01ffc48>] (warn_slowpath_null+0x2c/0x34)
[ 13.856781] r8:00000000 r7:00000000 r6:c18b8194 r5:60000093 r4:ef182000
[ 13.863708] r3:00000009
[ 13.866485] [<c01ffc1c>] (warn_slowpath_null+0x0/0x34) from [<c0237d84>] (check_flags+0x78/0x1d0)
[ 13.875823] [<c0237d0c>] (check_flags+0x0/0x1d0) from [<c023afc8>] (lock_acquire+0x4c/0x150)
[ 13.884704] [<c023af7c>] (lock_acquire+0x0/0x150) from [<c06af638>] (_raw_spin_lock+0x4c/0x84)
[ 13.893798] [<c06af5ec>] (_raw_spin_lock+0x0/0x84) from [<c01f9a44>] (sched_ttwu_pending+0x58/0x8c)
[ 13.903320] r6:ef92d040 r5:00000003 r4:c18b8180
[ 13.908233] [<c01f99ec>] (sched_ttwu_pending+0x0/0x8c) from [<c01f9a90>] (scheduler_ipi+0x18/0x1c)
[ 13.917663] r6:ef183fb0 r5:00000003 r4:00000000 r3:00000001
[ 13.923645] [<c01f9a78>] (scheduler_ipi+0x0/0x1c) from [<c01bc458>] (do_IPI+0x9c/0xfc)
[ 13.932006] [<c01bc3bc>] (do_IPI+0x0/0xfc) from [<c06b0888>] (__irq_usr+0x48/0xe0)
[ 13.939971] Exception stack(0xef183fb0 to 0xef183ff8)
[ 13.945281] 3fa0: ffffffc3 0001500c 00000001 0001500c
[ 13.953948] 3fc0: 00000050 400b45f0 400d9000 00000000 00000001 400d9600 6474e552 bea05b3c
[ 13.962585] 3fe0: 400d96c0 bea059c0 400b6574 400b65d8 20000010 ffffffff
[ 13.969573] r6:00000403 r5:fa240100 r4:ffffffff r3:20000010
[ 13.975585] ---[ end trace efc4896ab0fb62cb ]---
[ 13.980468] possible reason: unannotated irqs-off.
[ 13.985534] irq event stamp: 1610
[ 13.989044] hardirqs last enabled at (1610): [<c01c703c>] no_work_pending+0x8/0x2c
[ 13.997131] hardirqs last disabled at (1609): [<c01c7024>] ret_slow_syscall+0xc/0x1c
[ 14.005371] softirqs last enabled at (0): [<c01fe5e4>] copy_process+0x2cc/0xa24
[ 14.013183] softirqs last disabled at (0): [< (null)>] (null)
Signed-off-by: Ming Lei <ming.lei@canonical.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
32bit and 64bit on x86 are tested and working. The rest I have looked
at closely and I can't find any problems.
setns is an easy system call to wire up. It just takes two ints so I
don't expect any weird architecture porting problems.
While doing this I have noticed that we have some architectures that are
very slow to get new system calls. cris seems to be the slowest where
the last system calls wired up were preadv and pwritev. avr32 is weird
in that recvmmsg was wired up but never declared in unistd.h. frv is
behind with perf_event_open being the last syscall wired up. On h8300
the last system call wired up was epoll_wait. On m32r the last system
call wired up was fallocate. mn10300 has recvmmsg as the last system
call wired up. The rest seem to at least have syncfs wired up which was
new in the 2.6.39.
v2: Most of the architecture support added by Daniel Lezcano <dlezcano@fr.ibm.com>
v3: ported to v2.6.36-rc4 by: Eric W. Biederman <ebiederm@xmission.com>
v4: Moved wiring up of the system call to another patch
v5: ported to v2.6.39-rc6
v6: rebased onto parisc-next and net-next to avoid syscall conflicts.
v7: ported to Linus's latest post 2.6.39 tree.
> arch/blackfin/include/asm/unistd.h | 3 ++-
> arch/blackfin/mach-common/entry.S | 1 +
Acked-by: Mike Frysinger <vapier@gentoo.org>
Oh - ia64 wiring looks good.
Acked-by: Tony Luck <tony.luck@intel.com>
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
This patch makes TTBR1 point to swapper_pg_dir so that global, kernel
mappings can be used exclusively on v6 and v7 cores where they are
needed.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Commit 228e548e (net: Add sendmmsg socket system call) added the new
sendmmsg syscall. Add this to the syscall table for ARM.
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
