Experimental Physics and
Industrial Control System

Rod Nussbaumer <[email protected]> · Wed, 07 Jul 2010 08:07:13 -0700

[email protected] wrote:

I expect you meant to CC the list!
From: Rod Nussbaumer [mailto:[email protected]]

Michael Abbott wrote:

On Tue, 6 Jul 2010, Rod Nussbaumer wrote:

I'm trying to get EPICS running on an ARM CPU. So far I've got a build
of EPICS (3.14.10 & 3.14.11) against a glibc 2.3.6 and Linux 2.6.21,
where the only thread model supported is linuxthreads (not POSIX/nptl).
When I run an IOC, the iocShell crashes (segfault), although the rest of
the IOC (asyn device support), and record processing seem to stay
running. Is this a reason to suspect the tread model? I see in the EPICS
build commandline, the switch '-D_POSIX_THREADS'. Does this mean a
strict POSIX thread model is required?
The iocShell crashes right after iocInit. Running under gdb, I set a
breakpoint on iocInit. I see:

(gdb) step
Single stepping until exit from function iocshBody,
which has no line number information.
epics>
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 114696 (LWP 4439)]
0x0006be60 in ca_flush_io ()

I wouldn't suspect the thread model here.
I have successfully run EPICS on two ARM system architectures, old glibc
with linuxthreads and old ABI

Thank you. That is the architecture I'm using. I will pursue other
avenues to debug this. I'm still trying to track down the precise point
of failure. It does seem to be completely repeatable and consistent. I
haven't ruled out a compiler or C library issue; either a bug or
compatibility problem. When building EPICS base for ARM, I do get a lot
more warnings than I do with the x86 host-based build.
Did you develop your own root filesystem for ARM? What
toolchain are you using? Did you create that in-house?
What is required to create support for ARM (or other architecture) as an
EPICS base build host (EPICS_HOST_ARCH=linux-arm??). Has anyone already
done this?

Originally I used a toolchain provided by our vendor based on gcc 3.3.3 and glibc 2.3.2 and targeting the original linux-arm architecture (linuxthreads, old ABI). At that time we submitted patches to EPICS base to add a linux-arm architecture to the set of EPICS cross compilation architectures:
 CROSS_COMPILER_TARGET_ARCHS += linux-arm
This comprised files CONFIG.linux-x86.linux-arm and allies in configure/os in EPICS base, designed to be built using EPICS_HOST_ARCH=linux-x86 -- I've never tried to natively compile on ARM, as the target system is rather small, I imagine that 64MB RAM is a little tight for gcc these days!
One complication in the linux-arm architecture is that the floating point format (implemented by the Netwinder Floating Point emulation library, NWFP) for doubles is not IEEE format on a little-endian architecture (which I presume you're using), so this requires a special hack in the network format adapter of EPICS. You're almost certainly using this, and one of the huge advantages of EABI is that you can throw NWFP straight in the bin.

Yes, I'm using that. Old ABI, NWFP, little-endian.
Subsequently however I have put together a complete build system in-house. This was rather more work than expected, but at least the result is quite nice and should be reusable if you're interested. Putting together a complete system from scratch involves:
1. Compiler, toolchain and libraries. I recommend crosstool-ng if you are willing to build your toolchain from scratch -- it works very nicely and allows you to precisely target the configuration of your final system. I have detailed instructions and wrapper scripts. This toolchain is supported by the linux-arm_el EPICS target architecture which differs essentially in not requring the NWFP hack.

Yes, I'm using that, too. I've built my toolchain using CT-NG 1.5.3, the
newest version that still supports my kernel and glibc. Do you have a
.config file you'd be willing to share?
2. Bootloader (or you can just use your existing bootloader if it's up to scratch). I think all I did was enable the hush shell on u-boot using the vendor's original u-boot patches. This will be *very* target system dependent -- what *is* your target processor? ARM has a surprising number of variations.

I'm using a Technologics-Systems TS-7350 SBC with their custom
bootloader, booting from SD flash, but aim to change that to network
boot. For now, booting from flash is fine. Actually, a requirement for
the first deployment. The CPU is a Cirrus EP9302.

3. Kernel. I'm sure your 2.6.21 kernel is recent enough to support NPTL, but you may need to check that EABI support is compiled into the kernel (if you care; we did, I recommend it highly).

That was to be a longer term plan, and hoping to avoid it right now.
EABI seems too attractive to not use it, eventually. The kernel I'm
using includes binary-only modules for vendor-custom hardware, so
upgrading isn't just a kernel build. Running

'getconf GNU_LIBPTHREAD_VERSION' on my target says I have linuxthreads.

4. Finally assembling the target rootfs, which in our case is glibc + busybox + ntp + a handful of other utilities. I have a configurable (and somewhat overcomplicated) collection of build scripts and makefiles to automate all this, which you can try if you're interested...

Yes, I'm interested. I've tried buildroot, openembedded, and a couple of
others. Again, in the near term I plan to use the root filesystem
provided by the vendor. Eventually, though, I want to use something
smaller and simpler, and tailored to our requirements.

However, none of this is relevant to your original question! Can you simplify your IOC to nothing but an IOC shell, and does that still crash?
Not sure about that. I do have a simple IOC that has only soft-channel
device support, and loading a single CALC record as a database. The IOC
shell only crashes after running iocInit, so I don't think an
iocShell-only image would reproduce the fault (and I'm not sure how to
build such a thing). I have rebuilt EPICS base with debugging support
turned on, and have been trying to step through it to the point of
failure. I think the crash happens in a different thread than the
debugger is looking at, and occurs sort of asynchronously. Still working
through it.
Thanks for your help.
 --- rod.

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System