Step 2 - Extract from Makelog Approach

Under the Extract from Build Log approach, Imagix 4D pulls information from a log of the command line invocations issued for your compiler as it compiles your source code. Some of this same information, in particular the names of the source files and their associated -I, -D and -U arguments, is used by Imagix 4D's source analyzer to analyze your code.

There are two alternatives for creating such logfiles:

Command Capture: You may be able to generate a logfile directly as part of your build process. For example, as you run your make system, you might redirect the commands it echoes into a logfile. Or you may be able to set your compiler to run in verbose mode and to capture the echoed compiler commands to a logfile.

Compiler Monitor: Alternatively, you can use Imagix 4D's imagix-cc-monitor utility to record invocations of the compiler. The utility monitors system level processes. By setting the utility to identify the compiler(s) that you're using with your source code, all the compiler invocations can be recognized and captured to a logfile. See Appendix C for more detail about running imagix-cc-monitor.

The Extract from Build Log process uses the extracted information to populate the dialog-based data source definitions, so that the definitions precisely match how your code is compiled. At that point, you're able to modify the settings extracted from the logfile to fine tune how the Imagix 4D project is generated.

2a. Generate a logfile that contains the compiler commands

Note: This step 2a applies if you're using the Command Capture alternative. For the Compiler Monitor alternative, see Appendix C for details about running imagix-cc-monitor utility -- the resulting logfile will contain all the necessary data.

The Extract from Build Log approach requires that you have a log file containing the commands your build system issues as it compiles your source code. The most important of these commands are those issued to your compiler. Also required are commands causing a change of directory, so that absolute paths can be accurately calculated from any relative paths passed to the compiler.

The actual process involved in creating such a logfile varies significantly between build environments, so only the theory is presented here. When make is run, each command line required for a build is echoed to standard out and then executed. The general approach for generating a logfile is to invoke the make that compiles your code and to capture or redirect the echoed commands into a file. If you are currently running make with -silent or -quiet type options, you will need to disable those so that the make commands are echoed and can be logged. There may be additional options you need to add to cause make to echo the commands it is executing.

Once you've created the logfile, you'll need to confirm that it contains all of the elements required by Imagix 4D to analyze your code. The logfile will need to specify all of the the source files being compiled. For each of the compile commands, there will need to be an indication of the include directories and macro definitions used.

Here's an example of the essential elements required in a build log's compile command.

compiler -I../dir1 -Dmacrodef sourcefile.c

In the following small sample, as it might actually appear in a logfile, each of the essential elements exists.

make[2]: Entering directory `/home/fredn/projects/uclib/armv-gnueabi/drivers/power'
/usr/bin/gcc -g -Os -fno-strict-aliasing -fno-common -ffixed-r8 -msoft-float
    -D__KERNEL__ -DTEXT_BASE=0x80e80000 -I/home/fredn/armv-gnueabi/u-boot/include
    -isystem /home/fredn/armv-gnueabi/toolchain/lib/gcc/armv-gnueabi/4.4.5/include
    -fno-builtin -nostdinc -DCONFIG_ARM -D__ARM__ -march=armv5 -mno-thumb-interwork
    -Wall -Wstrict-prototypes -fno-stack-protector -o iex_power.o iex_power.c -c
make[2]: Leaving directory `/home/fredn/projects/uclib/armv-gnueabi/drivers/power'

You can see that buried within a lot of additional options, this sample contains all of the essential elements. The actual compilation command occurs on the second line, identified by the leading /usr/bin/gcc. There is a source file iex_power.c specified, as well as some -I include directory options and some -D macro definition options.

The first and third lines are also important, as they indicate where the source file is located. When the compilation command uses relative rather than absolute paths for either the source file or the include directories, some way to derive the actual directory location is necessary.

Typically, a logfile contains a large number of commands generated by the make system. The Extract From Makelog process acts as a filter, extracting the information about source files and their related options, along with the directory context. Imagix 4D then uses this information to populate the project settings with a series of data sources, each specifying a set of source files that share the same include directory and macro definition options.

2b. Specify that you're adding a new data source

Open the Data Sources dialog (Project > Data Sources). To specify that you wish to add a new data source into the project rather than modify the settings for an existing data source, choose `+ new data source' under the Data Sources selector on the left side of the dialog. As this is a new project, and there are no existing data sources to modify, this is indeed the only choice available.

2c. Select the extracting from build log approach

For the rest of step 2, you'll be working on the right side of the Data Sources dialog. At the top, select [Source Files][Make Based][Extract from Makelog] in the menubutton labeled `Select Data Source Type'.

You'll later see that this extract from build log approach defines a series of additional data sources on the left side of the dialog. In this process, any existing data sources are lost. To prevent this from occurring accidentally, the Extract from Build Log selection is only available when no other data sources have yet been defined.

There may be data sources you want to add to your project in addition to the ones defined through this approach. If so, it's best to create a separate project containing those additional data sources and to use the ComboProject feature to combine the two projects. (See ComboProjects for more info).

2d. Specify the name of your logfile

In the `Build log file' field of the Build Log tab, enter the full path/file name of your log file.

2e. Specify any additional compiler and source analyzer options

Normally, you will leave the Options field empty. All of the -I, -D and -U compiler flags that appear in the logfile will be automatically extracted. However, if your logfile is missing some of the compiler flags you would like to use while analyzing your source code in Imagix 4D, you can add them here. You can also add any Imagix 4D source analyzer options you want to apply. (See Analyzer Syntax and Options for more info).

2f. Select your compiler configuration file

In the Compiler & Target combobox, select the compiler configuration file that you set up in step 1. If you haven't yet configured a compiler configuration file for the compiler and target platform of your software, strongly consider doing so now. The alternative is to select `other'. If you select `other', you'll need to specify the implicit flags in the Options field (step 2e).

2g. Review and modify the processing rules

On the Processing Rules tab, you will find a series of rules controlling how your logfile is analyzed and how information is extracted. By reviewing the rules side-by-side with your logfile, you'll be able to specify rule settings that are compatible with the format of the logfile.

Note that this is the beginning of an iterative process. In step 2i, you will examine the results of processing the logfile using these rules, and then return to this step to refine your rules. You will be able to repeat this until you are satisfied with your results.

The Processing Rules tab is divided into a number of sections.

When each compiler command is processed, those tokens in the command line that end in one of the specified suffixes are considered to potentially be the name of a source file. Header files suffixes do not need to be specified; like a compiler, Imagix 4D analyzes the source files, and then pulls in whichever header files are included by the source files themselves.

In processing the logfile, the filter needs to identify which lines refer to compiler commands. The settings displayed here would cause the second line in the small sample above to be identified as a compiler command. This is because the /usr/bin/gcc (ignoring its path) at the beginning of the second line matches one of the compiler commands listed here.

If the name of the compiler in the logfile has a suffix, such as cl.exe, you'll need to include the suffix on the compiler command listed here. The two setting alternatives provide additional support for identifying compiler command lines in more complex logfiles.

The actual format of a compiler command can be vary widely depending upon the make system being used and the compiler being invoked. A number of settings are available to configure how the filter analyzes the logfile line once it has been identified as a compiler command.

Many Windows compilers use "/I directory" and "/D macro" rather than "-Idirectory" and "-Dmacro" style options. The second Format choice enables such compiler command options to be extracted from the logfile and converted to the equivalent Imagix 4D option.

The filter processes each token in the compiler command line. By knowing which options take arguments, the filter is able to skip over a meaningless argument and to avoid the further steps of determining whether it is another option or the name of a source file.

Make systems and compilers also differ in how they interpret and communicate quotation marks in the value term of a -D macro definition. The Imagix 4D source analyzer will include any quotation marks to the right of the = as part of the string substitution; if this is not appropriate, the excess quotations marks can be eliminated.

Note: The Logfile Processing settings apply if you're using the Command Capture alternative. If your logfile was generated through the Compiler Monitor alternative, the logfile format is known, the directory location is recorded and so the settings are ignored.

Additional settings are available for general handling of the logfile. Entire lines can be skipped if they start with a specified token. In logfiles whose long lines have been split via line returns, the broken lines can be concatenated back together without the need for manual intervention. In logfiles where a single physical line is made up of two or more logical lines joined by "&&", those logical lines can be analyzed as separate lines during the processing.

Often, the source files and -I include directory options in the logfiles use relative path names. In such cases, the logfile analysis needs to track the directory location associated with each compiler command. The filter treats logfile lines containing "Entering directory", "Leaving directory", "START" and "END" as indicating a change in directory location. Additional change directory commands can also be specified, as "cd" is here. The settings also provide control over how those change directory commands are processed.

In this directory tracking, the directory location where the logfile exists is considered to be the starting directory. You might choose to manually prepend a change directory command, such as "Entering directory c:\starting\dir", to the beginning of the logfile to make the starting directory explicit.

The Extract From Makelog process generates a separate data source for each set of source files having a different set of associated -D options. You can control whether this is further subdivided into separate data sources for each set of -I options, or if the -I options are combined for all of the source files sharing the same -D options.

When it encounters an option that it doesn't recognize, Imagix 4D's source analyzer converts it into a macro definition. For example, the option "-opt" gets converted to "-D__IMGX_opt__". The resulting macro definition can be used to control logic in the compiler configuration files. A filter setting enables you to control whether such unrecognized options get passed to the source analyzer.

If you've generated your logfile on another computer or even another operating system, you can use Imagix 4D's path substitution to modify the directory paths, so that source files and include directories are specified according to where they exist in your current environment.

2h. Process the logfile

At this point you're ready to process the logfile. Click Process Makelog at the bottom of the dialog.

2i. Review your results and refine the processing rules

Imagix 4D provides several sources of feedback to use in reviewing the results of processing the logfile.

The first indicator you'll see is a dialog that lists the number of source files that were identified during the processing, divided into those found in your file system and those that weren't. The dialog enables you to generate project settings. If you choose to, individual data sources are defined in the list on the left side of the Data Sources dialog. By clicking on those, you can review which source files were identified, as well as what -I, -D and -U flags were associated with each set of files.

Especially useful are the extraction logs generated while processing the logfile. These logs are a series of files titled extract_xxx.log, located Imagix 4D project directory (step 4a). Separate logs cover the compiler actions, the compiler details, the change directory actions, and process warnings. These logs can be accessed through the Status tab that appears on the right side of the Data Sources dialog once you have processed the logfile.

Reviewing the results and revising the rules is an iterative process. It will require analysis and deductive reasoning on your part. Here are some strategies that can make the process easier.

Identify source files

When first starting, focus on identifying the source files referenced in the logfile.

If no source files are being found, refer to the Compiler log. It shows which lines from your logfile have been identified as containing compiler invocations. If no compiler invocations are found, first examine your logfile. If you're unable to manually identify compiler invocations there, you may need to regenerate your logfile (step 2a). If compiler invocation lines exist there, but aren't being identified according to the Compiler log, check your Compiler Invocation rules. Make sure that the compiler command you have specified matches the command as it appears in the logfile. You may also need to revise the `Commands to ignore at the beginning of line' rule.

However, if compiler invocations are found and source files are still not being found, compare the suffixes in the invocations with your `Source file suffixes' rule.

Identify compiler options

Once the source files are being properly identified, review the extraction of compiler option flags.

Examining the Compiler Details log is particularly useful for this. You're able to compare the actual line from the logfile with the options that have been derived from it. If you see that options are being missed, or that the format of the options is not what you want, modify the Compiler Options rules.

You may also find the Warning log to be of help. Any `unrecognized arg' listings for a compiler line could indicate either that you need to modify the option format or that some options are missing from the `Compiler options taking argument' rule.

Adjust paths

As a last step, review and resolve the paths.

In addition to identifying the names of source files, and their associated -I, -D and -U flags, the extraction process tracks directory location, so that relative path names to both source files and include directories can be resolved. If you're using the Compiler Monitor alternative, the directory location is automatically recorded, but with the Command Capture alternative, you might need to make some adjustments to the settings in the Logfile Processing section.

The CD log provides insight into which lines in the logfile are identified as changing the base used for subsequent relative directory calculations. If cd invocation lines are being missed, review your `Change directory command' rule. If the cd invocation includes relative paths, select a `Processing change directory commands with relative paths' rule that causes the appropriate path to be calculated.

If the Summary of build log processing still indicates that other source files are being identified than those found in the file system, you can refer to the Warnings log to determine which files are not found in the file system. You'll then need to determine whether this is because the file is actually missing or because the file path was not properly resolved in the processing. If the latter occurred, the change directory rules need to be revised.

One final note for both alternatives with regard to paths. You may be in a situation where your source code is located in a different location for analysis by Imagix 4D than it was at the time you generated the logfile. This could occur for a number of reasons, including using a different platform for analysis than for compiling. If the logfile contains relative rather than absolute path names, you may have success copying the logfile from the compile environment to the same position relative to the analysis code, and then specifying the new copy as the logfile to process. Alternatively, if the logfile contains absolute paths, you can make path substitutions using the Directory Mapping settings in the Data Collection section of the File > Options dialog.

2j. Start the analysis process

Reviewing the results and revising the rules is an iterative process. Each time you modify the rules (step 2g) and process the logfile (step 2h), you'll generate fresh results to review. Initially, you'll see enough evidence in your review (step 2i) that it will be clear that you need to again refine the rules and (re)process the logfile.

At some point point, you'll decide that the processing rules appear good enough that you'd like to see how well the resulting Dialog-Based (C/C++) data source definitions analyze your source code. This is not an irrevocable step. You can later return to step 2g, or even 2a, and continue with the iteration inherent in step 2.

When you're ready to apply the generate Dialog-Based (C/C++) data source definitions to analyze your code, click Load Data Sources at the bottom of the dialog.