Experimental Physics and
Industrial Control System

Guy Jennings <[email protected]> · Mon, 10 Sep 2012 19:23:04 -0500

Might be worth taking a look at the Intel Thread Profiler for Linux -

I've used the Windows version and it is excellent

at showing threading problems. It's quite expensive, but there is a

30 day evaluation version.

On Sep 10, 2012, at 6:03 PM, Mark Rivers wrote:

Thanks for the suggestion, perf record and perf report work fine.

The results are interesting. 10% of the time is spend in the

pthread_mutex code.

 5.38% quadEMTestApp libpthread-2.14.1.so [.]

__pthread_mutex_unlock_usercnt

 `

 4.34% quadEMTestApp libpthread-2.14.1.so [.]

pthread_mutex_lock

 a

 2.36% quadEMTestApp quadEMTestApp [.]

getAddr

 a

 2.27% quadEMTestApp quadEMTestApp [.]

__i686

.get_pc_thunk

.bx

 a

 2.17% quadEMTestApp [kernel.kallsyms] [k]

_raw_spin_lock

 a

 1.98% quadEMTestApp quadEMTestApp [.]

paramList::int32Callback(int,

int

) a

 1.95% quadEMTestApp libpthread-2.14.1.so [.]

pthread_cond_signal

@@GLIBC_2.3.2

 a

 1.95% quadEMTestApp [kernel.kallsyms] [k]

ia32_syscall

 a

 1.79% quadEMTestApp [kernel.kallsyms] [k]

__audit_syscall_exit

 a

 1.55% quadEMTestApp quadEMTestApp [.]

epicsMutexLock

 a

 1.53% quadEMTestApp [kernel.kallsyms] [k]

resched_task

 a

 1.47% quadEMTestApp quadEMTestApp [.]

epicsMutexOsdLock

 a

 1.44% quadEMTestApp [kernel.kallsyms] [k]

do_futex

 a

 1.41% quadEMTestApp quadEMTestApp [.]

paramList

::getParameter

(int

) a

 1.35% quadEMTestApp [kernel.kallsyms] [k]

__schedule

 a

 1.29% quadEMTestApp quadEMTestApp [.]

getTraceMask

 a

 1.29% quadEMTestApp [kernel.kallsyms] [k]

update_cfs_shares

 a

 1.24% quadEMTestApp [kernel.kallsyms] [k]

update_curr

 a

 1.24% quadEMTestApp quadEMTestApp [.]

portThread

 a

 1.23% quadEMTestApp quadEMTestApp [.]

epicsMutexUnlock

 a

 1.23% quadEMTestApp [kernel.kallsyms] [k]

native_write_msr_safe

 a

 1.20% quadEMTestApp [kernel.kallsyms] [k]

select_task_rq_fair

 a

 1.20% quadEMTestApp [kernel.kallsyms] [k]

try_to_wake_up

 a

 1.12% quadEMTestApp [kernel.kallsyms] [k]

update_rq_clock

 a

 1.12% quadEMTestApp [kernel.kallsyms] [k]

futex_wait

 a

 1.09% quadEMTestApp [kernel.kallsyms] [k]

update_cfs_load

 a

 1.08% quadEMTestApp [kernel.kallsyms] [k]

device_not_available

 a

 0.92% quadEMTestApp quadEMTestApp [.]

drvQuadEM

::computePositions

(int

*) a

 0.91% quadEMTestApp [kernel.kallsyms] [k]

hash_futex

 a

 0.91% quadEMTestApp [kernel.kallsyms] [k]

futex_wake

 a

 0.89% quadEMTestApp [kernel.kallsyms] [k]

plist_add

 a

 0.87% quadEMTestApp [kernel.kallsyms] [k]

ia32_sysenter_target

 a

 0.83% quadEMTestApp [vdso] [.]

0xf77ac421

 a

 0.83% quadEMTestApp libpthread-2.14.1.so [.]

pthread_cond_wait

@@GLIBC_2.3.2

 a

 0.81%  quadEMTestApp  [kernel.kallsyms]     [k] effective_load

Mark

-----Original Message-----
From: J. Lewis Muir [mailto:[email protected]]
Sent: Monday, September 10, 2012 5:37 PM
To: Mark Rivers
Cc: [email protected]
Subject: Re: How to profile an EPICS application on Linux

On 9/10/12 5:21 PM, Mark Rivers wrote:

Here is the output of "time" when running it in a mode where "top"

showed the application using about 95% of a single CPU (on a 16

core system). This is with my calculations enabled.

23.547u 16.516s 0:52.39 76.4%   0+0k 0+24io 0pf+0w

This is the output of time when the calculations are disabled and

top shows about 80% CPU in my application.

Disabling my calculations:
11.401u 8.008s 0:27.60 70.2%    0+0k 0+8io 0pf+0w

So it spends a substantial amount of CPU time in both User Mode and

Kernel Mode.

Hi, Mark.

OK, then I don't know what's going on.  Maybe gprof has problems
with threaded programs (??).

I'm really not answering your question now, but another idea: Is
your Linux system new enough that you have the perf subsystem
available?  You could try that instead:

$ perf record ../../bin/linux-x86/quadEMTestApp
$ perf report

Lewis

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System