Troubleshooting

Common issues

This page provides inputs to resolve common issues seen during developement.

make is pulling the Hexagon Tools from the wrong location

The environment variable "HEXAGON_TOOLS_ROOT" points to the default location for Hexagon Tools which is set to ${HEXAGON_SDK_ROOT}/tools/HEXAGON_Tools/

You can override the default tools location by setting the environment variable HEXAGON_TOOLS_ROOT to the desired version of the Hexagon Tools.
How to fix undefined PLT symbol errors seen during execution on the Hexagon simulator or the device?

While running the application, this error is seen when any of the loaded libraries are unable to find the definition of the symbol in the set of loaded libraries, i.e the skeleton libraries and their dependencies. Hence, ensure that the symbol is defined in one of the libraries given as input to the linker before pushing the libraries to the target. To find out the symbols in a library, use Hexagon Tool hexagon-llvm-readelf as follows:
```
%DEFAULT_HEXAGON_TOOLS_ROOT%/Tools/bin/hexagon-llvm-readelf -a <library_path>
```
Look out for the symbol in this output to figure out if the symbol is defined or not (UND). Use the linker option -u or -extern-list to provide the symbol name as argument in the linker command to explicitly export the symbol in the output library. Refer to linker doc for complete details.
How to resolve the following error while building my library?
```
undefined reference to `stdout@LIBC'
undefined reference to `stderr@LIBC'
```
These symbols are part of libc library and this error is seen when your library has these symbols and libc is not linked to your library. Link libc to your library to reslove this error.

e.g: If you are getting this error libitrace.so: undefined reference to 'stdout@LIBC'

Add -L$(HEXAGON_SDK_ROOT)/tools/android-ndk-r19c/platforms/android-26/arch-arm64/usr/lib/ -lc in your linker command to resolve the error.
Why is the Hexagon simulator hexagon-sim crashing?

If you work with Windows, please make sure NOT to have the Hexagon tools as part of the path specified by the PATH environment variable.
What should I do if I see the following error while building my application?
```
dlerror undefined symbol __gxx_personality_v0 in ./<user_lib>.so)
```
The symbol __gxx_personality_v0 is part of libc++ and libc++abi libraries. The error is seen when these libraries are improperly linked with the user library. The libc++ standard library is distributed as two distinct libraries, libc++ and libc++abi. When -stdlib=libc++ is used, the compiler/linker is configured to link these two libraries automatically. If -nostdlib argument is used to disable the automatic inclusion of libc++, then the C++ code will require linkage with both libc++ and libc++abi. Hence, the issue can be resolved by linking both libc++ and libc++abi libraries with the user library.
What should I do if I see the following error while running my application?
```
Fatal: Current link configuration does not support relocation type `R_HEX_32_6_X' for symbol `_ZSt4clog' referred from C:/Qualcomm/Hexagon_Tools/8.3.04/Tools/target/hexagon/lib/v66/G0/libstdc++.a(iostream.o)[.text] , recompile with -fPIC hexagon-clang.exe: error: hexagon-link command failed with exit code 1 (use -v to see invocation)
```
This error is seen if libstdc++ is linked statically with the user library compiled as a shared object. The library libstdc++ residing in the pic folder has to be linked with the user shared object. For example, libstdc++ is to be picked from the directory %DEFAULT_HEXAGON_TOOLS_ROOT%/Tools/target/hexagon/lib/v66/G0/pic.
Why is dynamic loading failing?

This happens usually when the shared object is not found in the search path specified with [A]DSP_LIBRARY_PATH. Refer to Remote file system. To know about DSP messages during dynamic loading failure, refer to Trouble shooting fastRPC issues section
Why does my FastRPC call not return?

Capture and analyze the DSP logs with mini-dm or logcat. If you do not see any logs from your own application, do you see log messages suggesting that your device is not signed, that your shared object cannot be found, or that no shared object is implementing the DSP function you are trying to call?
What should I do if I notice high latency for RPC calls?

Follow these steps to handle high FastRPC latency:
- The primary boot image adds more than 1 ms overhead to a FastRPC call by enabling extra logging in kernel modules. For best performance, use a secondary boot image by either working with a production device or asking your CE on how to flash a secondary boot image.
- Always use ION buffers in any remote call. Buffers allocated in heap or stack would add an overhead due to extra memory copy. For more details refer RPCMEM APIs
- Ensure voting for appropriate/max clocks on ARM & DSP as shown in the profiling benchmark.
- Skip first RPC call overhead. First call includes session open latency.
- Find average, minimum and maximum overhead over N iterations
- Enable FastRPC QoS using the remote APIs
Using incorrect array size with FastRPC

When passing an array to the DSP using FastRPC, the array size must be specified as the number of elements in the array, not as its size in bytes.
Missing the 128-byte compilation flag

If you compile a function to be run with the 128-byte HVX mode, you must explicitly set the compilation flag.
```
-mhvx-length=128B
```
Using incorrect memory alignment for scalar accesses
- in C programming, make sure you consistently use the correct data type for all memory allocations and memory de-references.
- in Assembly programming, make sure you use the correct mem instruction (e.g. memb, memh, memw, memd) for the desired data width, and that the addresses that you are using, matches the same alignment.
Compiling for the wrong target

Ensure your compilation flag is compatible with the architecture you are targeting. For example, do not run code compiled for V69 on a Lahaina device.
Making out-of-bound memory accesses

Load and stores to unmapped/invalid addresses cause TLB-miss exceptions. Note that L2 prefetching also triggers a TLB-miss exception if the start address is unmapped/invalid in the calling PD. However, if a prefetch crosses into an invalid page, the remainder is simply dropped without failure.
Missing a function implementation

A missing function implementation does not result in a compilation error if the function is declared as extern but it results in a runtime dynamic linking error. Inspect the DSP logs for the presence of an undefined PLT symbol i.e. funtion_name in your_library.so message. Also, remember that C++ mangles function names, so to access a function written in C or assembly from a C++ file, the function must be defined as extern “C”.
Using an incorrect function declaration

If you modify the parameters of an implementation, remember to modify the parameters of its declaration. Otherwise, your code might still compile but you end up passing incorrect parameters to a function and get an unexpected behavior.
Modifying callee-saved registers

The calling conventions specify which registers are the responsibility of the calling function to save, and which registers are the responsibility of the callee function to save. This is described in the Hexagon Application Binary Interface User_Guide. When writing assembly code, it is important to save all callee-saved registers if you modify them in your implementation. These registers are R16-R27 and R29-R31. The following example shows how you do so with registers R16-R19, for example:
```
{
  ALLOCFRAME(#2*8)    // enough space for 4 32-bit registers
                      // Note: needs to be a multiple of 8
} {
  memd(r29 + #0) = r17:16
  memd(r29 + #8) = r19:18
} {
  <your code here may use and modify registers r16-r19 in addition to any caller-saved register>
} {
  r17:16 = memd(r29 + #0)
  r19:18 = memd(r29 + #8)
} {
  DEALLOC_RETURN
}
```
Failing to make assembly code position-independent

Code can compile properly when testing for the simulator, which builds a static library, but fail to compile when building a dynamic library. This might be caused by setting a register to an immediate address:
```
test_array:
    .byte 0,1,2,3
...
r1 = #test_array // absolute addressing will only compile properly when building a static library
r1 = ADD(PC,##test_array@PCREL) // PC-relative addressing is needed when building a dynamic library
```
Running out of memory on the stack

FastRPC created user thread stacks are 16 KB currently. Do not use excessive local variables, and be especially careful with HVX intrinsics, which might spill onto the stack and quickly consume large amount of stack memory.
When is make: Command not found error seen?

If you have already run the Hexagon SDK setup script, this error is seen when gow is not installed along with SDK installation. On Windows and after running the Hexagon SDK setup script, run the command which make in the console. If the Hexagon SDK is properly setup, this should point to the installed gow location.
```
which make
C:\Qualcomm\Hexagon_SDK\5.4.1.0\tools\utils\gow-0.8.0\bin\make.EXE
```
Is it possible to run an application from /data/app instead of /vendor/bin?

With the introduction of treble limitations starting with Android-P, it is not possible to run an application from /data/app. However, users can run the application from /data/nativetest/vendor or /data/nativetest64/vendor, depending on if it is a 32-bit or a 64-bit application. The executable can also be run from the location /data/local/tmp. An application run from any of these locations will have access to the libraries in the vendor partition.
What to do to resolve the following loader issue on the Hexagon simulator?
```
 _rtld_map_object_ex: cannot open libc++.so.1, errno 2 (no such file or directory)
```
This issue could be seen when the hexagon simulator executable is linked with the libraries libc++.so and libc++abi.so. These libraries have symbolic links as follows:
```
libc++.so -> libc++.so.1 -> libc++.so.1.0
libc++abi.so -> libc++abi.so.1 -> libc++abi.so.1.0
```
When the simulator loads the executable, it also tries to load the dependent c++ libraries and looks for their symbolic links during this process. These symbolic links have to be available in the current directory to be available to be loaded by the simulator. The solution to this issue is illustrated in hexagon.min as part of the calculator C++ example.
What to do when I see the following error message when running mini-dm?
```
Device open failed with error -3
```
This issue is seen if your device is not added to the udev rules. Please add your device to the udev rules. Only a user with sudo privilege can make changes to the udev rules. Alternatively, you can run mini-dm with sudo.
What to do when building C++ examples and the build fails when linking pthread.h?
```
fatal error: 'pthread.h' file not found
include <pthread.h>
error generated.
```
The header file pthread.h is included with the QuRT libraries. This issue is seen when a C++ based library is built as a standalone library without linking to QuRT. Refer to the calculator_c++ example for building the library with pthread.h from QuRT. Alternatively, if you do not want to link with QuRT, you can pass a compiler flag specifying a C++ standard, that has no build dependency on pthread.h such as C++03. The compiler flag can be passed to the hexagon.min file as follows:
```
CXX_FLAGS += -std=c++03
```
For CMake, the compiler flag can be passed to the CMakeLists.txt file as follows:
```
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++03")
```
Building examples using CMake fails due to error No rule to make target.
```
make: *** No rule to make target 'conv_utility_q'. Stop.
```
This error means that the target CMake is trying to build is not defined in CMakeLists.txt. Add the target definition in the CMakeLists.txt to resolve this error.

Eclipse IDE

Why do errors appear in the C/C++ editor even though the code looks OK?

There are three possible reasons for this:
- The text strings for the project name, workspace name, or SDK root may contain space characters. Remove any spaces that appear in these strings and try again.
- Indexing of the project may not have occurred properly. To fix this, right-click on the project and choose Index -> Rebuild.
Why does the build log state "No such file or directory" even though the file is available and accessible to a newly-created template project?

A possible reason for this error could be the presence of space characters in the text strings specifying the project path or Hexagon SDK Root location. Remove any spaces that appear in these strings and try again.
Why do I get build errors in a project after integrating a library?

This can happen if the pathname of the integrated library is incorrect:
- Check if you integrated the library using the same configuration it was built with. For example, if a library is built using the ReleaseG configuration, you cannot integrate the library using the Debug configuration.
- Check the linker settings in the build configuration to see if the library that was integrated is actually present.
While debugging or running a project, why aren't certain views visible?

Choose Show View from the Window menu and select the required view. If the view you want is not listed, select Other and then choose it from there.
When trying to start the debugger, why do I get the error "IDE taking too long to connect"?

This happens when the application cannot attach to the debugger because the debug port is already being used.

This can be resolved by changing the port number. Make sure that the ports in the application code match what is specified in the IDE configuration. Refer to target debugging using eclipse for more details
Why doesn't the debugger not break at a breakpoint?

Switch to the Debug perspective to see the execution stopped at the breakpoint.

Trouble shooting fastRPC issues

If you run into errors when offloading an application onto a Hexagon DSP, you should be gathering the logs generated by the DSP for more information. The rest of the page discusses some of the most common errors that you may encounter.

If the error you run into is not discussed on this page, please refer to $HEXAGON_SDK_ROOT/incs/stddef/AEEStdErr.h for a complete list of error codes.

Analyzing errors

Operation not permitted

Error <0xffffffff>: apps_dev_init failed. <domain 3>, errno Operation not permitted

This error is seen when FastRPC fails to open a session with DSP or fails to create a corresponding user process on DSP.
- Check whether the DSP is up and running by referring to Checking DSP State section.
If DSP is not up and running, then please contact Qualcomm customer engineering(CE) team.
- Check whether the file fastrpc_shell_<0/3> loaded on target is same as the one in the DSP image. You can check time stamps of these files.
  
  Location on the target for Android P onwards: /vendor/dsp/<adsp/cdsp>
  
  Location in the DSP image: <adsp/cdsp>_proc/build/ms/dynamic_modules/<chip_number>.<adsp/cdsp>.prod
  
  If timestamp doesn't match, then flash aDSP/cDSP image again properly. If you do not have access to the DSP image,contact Qualcomm customer engineering (CE) team.
- Check DSP logs for any errors during user PD creation. PD creation can fail because of file read failures on HLOS.
Check if the user process has the required SELinux policy enabled. Check for the Access Vector Cache(avc) denial errors in kernel logs.

Here is an example of denial errors you might see when attempting to access the /dsp folder:

dmesg: avc: denied { open } for pid=10222 comm='XXXXXXXXX' path='/dsp/fastrpc_shell_0' dev='sdd7' ino=26 scontext=u:r:xxxxxxxx:s0 tcontext=u:object_r:unlabeled:s0 tclass=file permissive=1 SEPF_SECMOBILE_7.0_0002

If above errors are found, check whether Selinux policy is defined properly for your process.

For example: For cameraserver process to be able to access a particular directory, the following entry should be there in "/device/qcom/sepolicy/vendor/common/cameraserver.te" file:

r_dir_file(cameraserver, adsprpcd_file); (#allow cameraserver to access/dsp)
Sigverify failure

The following messages indicate a signature verification failure error:
1. CDSP: signature verify start failed for <xxxx>_skel.so
This error is seen when a test signature testsig-<xxxx>.so for the target is not found in the designated location on the target. This file is needed to run unsigned dynamic shared libraries on the DSP outside an unsigned protection domain.

Please review the documentation about signing and check the following:
- Check whether testsig-.so is present on the paths specified by the [A]DSP_LIBRARY_PATH environment variable
- If testsig-.so is present on the DSP library path, then check whether the serial number for the device and the generated testsig match.
- If you do not want to use a test signature, please read more about the signing process to understand how to properly sign a single library object or run within an unsigned PD environment.
Exception in user process on DSP and the user process is killed (Error: 39/0x27 or Error 0x8000040d)

Error 0x8000040d: remote_handle_invoke failed for handle 0x3, method 4 on domain 3 (sc 0x4020200) (errno Success) or Error 27: remote handle invoke failed. <domain 3, handle 3, sc 4020200, pra 0x72b7cc13f8> AEE_ENOSUCH = 39 (0x27)

Further FastRPC calls will return the same error code 39. Clients are expected to handle this error code and restart the session as discussed here.
- You may also want to collect a crash dump to understand why this error occurred. Please refer to the debugging instructions for more information.
- Alternatively, you may want to use the remote debugger and step through your code to understand where your error occurs.
  
  This error may also be seen if you are using a shared object that is built for an architecture incompatible with the target on which your application is running.
  
  Call stack is printed when crash happens and user can try fixing the issue .
  
  The table below shows the latest Hexagon version supported by each target:
  
  Target Hexagon Version on CDSP
  
  Lahaina V68
DSP Restart Error A remote procedure call returns AEE_ECONNRESET when an SSR occurred and triggers a restart of the DSP. The user then needs to wait for the DSP to complete its restart and reconnect to all the HLOS services before relaunching their own session.

This approach is illustrated in calculator_test.c as part of the calculator example.
User Process failed to load the shared object on DSP

Error 80000406: remote handle open domain failed.<domain 2, name adspmsgd_adsp>, dlerror cannot open <xxxxx>_skel.so AEE_EUNABLETOLOAD

This error is seen when the process failed to load the shared object on DSP.
- Ensure the shared object files are present on target in the directories defined by [A]DSP_LIBRARY_PATH environment variable.
- Make sure that either testsig is present on the target or the library and all its dependencies are signed.
- Check the DSP logs logs for any dlopen errors. Most common reason for dlopen errors is Undefined PLT symbol
  
  dlopen error:<file:///libsysmondomain_skel.so?sysmondomain_skel_handle_invoke&_modver=1.0&_dom=sdsp> undefined symbol PLT
This error comes from DSP, when a symbol is used in shared object but not exposed in FastRPC shell or not found in any of the dynamically linked libraries.
- Ensure SELinux policy is properly defined so that the process has access to load the shared object
Error: 80000402. Failure due to not enough memory in FastRPC Heap

Error 80000402: remote handle invoke failed. <domain 3, handle b89040f0, sc 3020200, pra 0x85205698> AEE_ENOMEMORY = 0x80000402

This error indicates that there is not enough memory in FastRPC Heap to allocate memory for stack allocation and thread creation in the userPD on DSP.

You can fix this error, by modifying the stack size using remote_session_control API
Error due to mismatch in stub & skel Error: 8000040e

CDSP: Error:8000040e open_mod_table_handle_invoke failed for <handle 7d203f50 sc 8010000 0714> mod_table.c AEE_EBADPARM = 0x8000040e

These errors can be seen:
- when the parameters passed to the DSP function do not match those declared in its IDL definition.
- when the stub and the skel are not compiled against the same version of the IDL file.
Hence make sure that:
- The parameters match the function signature
- The right version of the shared objects for DSP and application are pushed onto the device at the right location specified
Fopen failure

Error 45: fopen failed for <xxxxx>_skel.so (No such file or directory)

If you see an fopen failure message in the DSP logs, then follow the recommendations below:
- Check that the file "_skel.so" is present on target in the directories defined by [A]DSP_LIBRARY_PATH environment variable
- Check whether Selinux policy is defined properly for your process as mentioned above.
For example: For cameraserver process to be able to access a particular

Untrusted application trying to offload to a signedPD on the DSP: Error AEE_ECONNREFUSED (0x72)

logcat: `Error 0x72: apps_dev_init: untrusted app trying to offload to signed remote process (errno 111, Connection refused). Try offloading to unsigned-PD using remote_session_control`

dmesg: `Error: adsprpc (3400): calculator_mult: fastrpc_init_create_dynamic_process: untrusted app trying to offload to signed remote process`

If calling "_open" to spawn a new process on cDSP and obtain its handle fails with the error above, then request for unsigned-PD offload using the remote_session_control API and retry the handle open call. The code below demonstrates that approach:

```
int err = 0;
char *cDSPURI = calculator_URI CDSP_DOMAIN;
remote_handle64 handle = 0;

/* Try to open a signed PD on cDSP */
err = calculator_open(cDSPURI, &handle);
if (err == AEE_ECONNREFUSED) {
    /*
     * Untrusted app has been denied offloading to a signed PD.
     * Request for unsigned-PD offload.
     */
    struct remote_rpc_control_unsigned_module data = {1, CDSP_DOMAIN_ID};
    err = remote_session_control(DSPRPC_CONTROL_UNSIGNED_MODULE, (void*)&data, sizeof(data));
    if (err) {
        // unsigned-PD offload request failed
        printf("Error 0x%x: unsigned-PD offload request using remote_session_control failed\n", err);
        goto bail;
    }
    // Retry handle open
    err = calculator_open(cDSPURI, &handle);
}
```

If your application cannot run as unsigned, work with your OEM to whitelist your application instead.

Below error messages that are seen during process exit can be ignored.

01-11 00:44:33.431 32220 32220 I : [Test Case] TestRandomGraph/SingleOperationTest.L2_POOL_2D_V1_2/49 BEGIN 01-11 00:44:33.587 949 31916 D : vendor/qcom/proprietary/commonsys-intf/adsprpc/src/fastrpc_apps_user.c:931: Error 0xffffffff: remote_handle_invoke failed for handle 0x3, method 4 on domain 3 (sc 0x4020200) 01-11 00:44:33.587 949 31916 E : vendor/qcom/proprietary/commonsys-intf/adsprpc/src/listener_android.c:244:listener protocol failure ffffffff 01-11 00:44:33.587 949 31916 D : vendor/qcom/proprietary/commonsys-intf/adsprpc/src/fastrpc_apps_user.c:931: Error 0x8000040d: remote_handle_invoke failed for handle 0x3, method 4 on domain 3 (sc 0x4020200) 01-11 00:44:33.587 949 31916 E : vendor/qcom/proprietary/commonsys-intf/adsprpc/src/listener_android.c:251::error: -2147482611: 0 == (nErr = __QAIC_HEADER(adsp_listener_next2)( ctx, nErr, 0, 0, &ctx, &handle, &sc, inBufs, inBufsLen, &inBufsLenReq)) 01-11 00:44:33.587 949 31916 E : vendor/qcom/proprietary/commonsys-intf/adsprpc/src/listener_android.c:333:Error 0x8000040d: listener2 thread exited 01-11 00:44:33.588 949 31916 D : vendor/qcom/proprietary/commonsys-intf/adsprpc/src/fastrpc_apps_user.c:947: Error 0x8000040d: remote_handle64_invoke failed for handle 0x8905cc90, method 3 on domain 3 (sc 0x3000000) 01-11 00:44:33.589 949 31917 E : vendor/qcom/proprietary/commonsys-intf/adsprpc/src/log_config.c:322:Received exit.
undefined symbol __cxa_thread_atexit_impl in libc++abi.so.1

CDSP:symbol.c:350:0x4167:2053: undefined symbol #257 __cxa_thread_atexit_impl in ./libc++abi.so.1

The symbol __cxa_thread_atexit_impl is introduced in Hexagon Tools version 8.6.x and is not exposed on DSP image of targets prior to waipio. If you see this error while running your application that uses libc++ libraries, please use the toolchain that is compatible with your target.
ERROR: "QAIC executable version to build skel library XXXXX must be >= YYYYY" in DSP logs

Example: ERROR: "QAIC executable version to build skel library 10042 must be >= 10043"

If the DSP log throws this error it means that skel library you are trying to load is not generated using a recent-enough version of QAIC (version >= 01.00.43). Please follow the steps mentioned below to regenerate your skel library:
- Either install 5.0.0 base SDK with its early-release addon or install a more recent SDK
- Running the environment setup script will show the QAIC version, which should be at least 01.00.43
```
C:\Qualcomm\Hexagon_SDK\5.0.0.0>setup_sdk_env.cmd
...
Qaic's Another IDL Compiler: version 01.00.43.
Copyright (c) 2012-2022 QUALCOMM Inc. All rights reserved.
...
Setup Done
```
- Regenerate your skel lib. For instance to get the libcalculator_skel.so follow steps listed below, then pick updated skel library from hexagon_Debug_toolv86_v73\ship\libcalculator_skel.so
```
cd examples\calculator
build_cmake hexagon BUILD=ReleaseG DSP_ARCH=v73 DOMAIN_FLAG=3 VERBOSE=0
```
- You can verify the QAIC version used to generate the skel lib by opening \*_skel.c file and searching for the string qaic_version:
```
_ATTRIBUTE_VISIBILITY uint32_t calculator_skel_handle_invoke_qaic_version = 10043;
```
A AEE_ESTUBSKELVERMISMATCH error returned from a FastRPC call means that the skel was generated with a version of an IDL file lower than that of the stub. Please refer to the description of the stub-skel mismatch check feature for more information.

Checking DSP state

You can check for state of DSP with the help of below commands:

Find the proper subsystem node for ADSP: adb shell cat /sys/bus/msm_subsys/devices/subsys<1/2/3/4>/name should show "adsp/cdsp".

Then run adb shell cat /sys/bus/msm_subsys/devices/subsys<1/2/3/4>/state to find whether it is "ONLINE".