g+
g+ Communities
Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014 
<== Date ==> <== Thread ==>

Subject: Re: How to profile an EPICS application on Linux
From: Guy Jennings <jennings@anl.gov>
To: Mark Rivers <rivers@cars.uchicago.edu>
Cc: tech-talk@aps.anl.gov
Date: Mon, 10 Sep 2012 19:23:04 -0500
Might be worth taking a look at the Intel Thread Profiler for Linux - I've used the Windows version and it is excellent at showing threading problems. It's quite expensive, but there is a 30 day evaluation version.

On Sep 10, 2012, at 6:03 PM, Mark Rivers wrote:

Thanks for the suggestion, perf record and perf report work fine.

The results are interesting. 10% of the time is spend in the pthread_mutex code.


5.38% quadEMTestApp libpthread-2.14.1.so [.] __pthread_mutex_unlock_usercnt ` 4.34% quadEMTestApp libpthread-2.14.1.so [.] pthread_mutex_lock a 2.36% quadEMTestApp quadEMTestApp [.] getAddr a 2.27% quadEMTestApp quadEMTestApp [.] __i686 .get_pc_thunk .bx a 2.17% quadEMTestApp [kernel.kallsyms] [k] _raw_spin_lock a 1.98% quadEMTestApp quadEMTestApp [.] paramList::int32Callback(int, int ) a 1.95% quadEMTestApp libpthread-2.14.1.so [.] pthread_cond_signal @@GLIBC_2.3.2 a 1.95% quadEMTestApp [kernel.kallsyms] [k] ia32_syscall a 1.79% quadEMTestApp [kernel.kallsyms] [k] __audit_syscall_exit a 1.55% quadEMTestApp quadEMTestApp [.] epicsMutexLock a 1.53% quadEMTestApp [kernel.kallsyms] [k] resched_task a 1.47% quadEMTestApp quadEMTestApp [.] epicsMutexOsdLock a 1.44% quadEMTestApp [kernel.kallsyms] [k] do_futex a 1.41% quadEMTestApp quadEMTestApp [.] paramList ::getParameter (int ) a 1.35% quadEMTestApp [kernel.kallsyms] [k] __schedule a 1.29% quadEMTestApp quadEMTestApp [.] getTraceMask a 1.29% quadEMTestApp [kernel.kallsyms] [k] update_cfs_shares a 1.24% quadEMTestApp [kernel.kallsyms] [k] update_curr a 1.24% quadEMTestApp quadEMTestApp [.] portThread a 1.23% quadEMTestApp quadEMTestApp [.] epicsMutexUnlock a 1.23% quadEMTestApp [kernel.kallsyms] [k] native_write_msr_safe a 1.20% quadEMTestApp [kernel.kallsyms] [k] select_task_rq_fair a 1.20% quadEMTestApp [kernel.kallsyms] [k] try_to_wake_up a 1.12% quadEMTestApp [kernel.kallsyms] [k] update_rq_clock a 1.12% quadEMTestApp [kernel.kallsyms] [k] futex_wait a 1.09% quadEMTestApp [kernel.kallsyms] [k] update_cfs_load a 1.08% quadEMTestApp [kernel.kallsyms] [k] device_not_available a 0.92% quadEMTestApp quadEMTestApp [.] drvQuadEM ::computePositions (int *) a 0.91% quadEMTestApp [kernel.kallsyms] [k] hash_futex a 0.91% quadEMTestApp [kernel.kallsyms] [k] futex_wake a 0.89% quadEMTestApp [kernel.kallsyms] [k] plist_add a 0.87% quadEMTestApp [kernel.kallsyms] [k] ia32_sysenter_target a 0.83% quadEMTestApp [vdso] [.] 0xf77ac421 a 0.83% quadEMTestApp libpthread-2.14.1.so [.] pthread_cond_wait @@GLIBC_2.3.2 a
 0.81%  quadEMTestApp  [kernel.kallsyms]     [k] effective_load

Mark


-----Original Message-----
From: J. Lewis Muir [mailto:jlmuir@imca-cat.org]
Sent: Monday, September 10, 2012 5:37 PM
To: Mark Rivers
Cc: tech-talk@aps.anl.gov
Subject: Re: How to profile an EPICS application on Linux

On 9/10/12 5:21 PM, Mark Rivers wrote:
Here is the output of "time" when running it in a mode where "top" showed the application using about 95% of a single CPU (on a 16 core system). This is with my calculations enabled.

23.547u 16.516s 0:52.39 76.4%   0+0k 0+24io 0pf+0w

This is the output of time when the calculations are disabled and top shows about 80% CPU in my application.

Disabling my calculations:
11.401u 8.008s 0:27.60 70.2%    0+0k 0+8io 0pf+0w

So it spends a substantial amount of CPU time in both User Mode and Kernel Mode.

Hi, Mark.

OK, then I don't know what's going on.  Maybe gprof has problems
with threaded programs (??).

I'm really not answering your question now, but another idea: Is
your Linux system new enough that you have the perf subsystem
available?  You could try that instead:

$ perf record ../../bin/linux-x86/quadEMTestApp
$ perf report

Lewis



Replies:
RE: How to profile an EPICS application on Linux Malitsky, Nikolay D
References:
How to profile an EPICS application on Linux Mark Rivers
Re: How to profile an EPICS application on Linux J. Lewis Muir
RE: How to profile an EPICS application on Linux Mark Rivers
Re: How to profile an EPICS application on Linux J. Lewis Muir
RE: How to profile an EPICS application on Linux Mark Rivers
Re: How to profile an EPICS application on Linux J. Lewis Muir
RE: How to profile an EPICS application on Linux Mark Rivers

Navigate by Date:
Prev: Re: camonitor bug for string as array of chars Andrew Johnson
Next: RE: How to profile an EPICS application on Linux Malitsky, Nikolay D
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014 
Navigate by Thread:
Prev: RE: How to profile an EPICS application on Linux Mark Rivers
Next: RE: How to profile an EPICS application on Linux Malitsky, Nikolay D
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014 
ANJ, 18 Nov 2013 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· EPICSv4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·