Friday, February 21, 2014

Software developer competency matrix

One of the ACCU mailing lists currently has a thread going where the challenges of hiring good C++ developers is being discussed. Someone posted a link to a software developer competency matrix. I thought it was rather good so decided to blog about so I didn't forget and so that I would have something to go back to for reference purposes.

Someone else in the thread posted that they had been inspired by another one but I found that was much more career/corporation oriented. The first one is much better for a techie to measure up against. It can be used to measure oneself or if you are involved the hiring game.

Tuesday, February 04, 2014

Reverse engineering a sybase database

The world seems to have decided on Oracle. But every now and then as a freelancer I come across a system that uses sybase. Usually it is an ancient legacy system that cannot be turned off for some reason, so sybase soldiers on. In such cases it is often useful to be able to reverse engineer the schema. This is where SchemaSpy comes in.

SchemaSpy requires graphviz, aka dot. The final generated output is a set of web pages and a set of dot files. I suspect that the dot is used to generate the image files used on the web pages, so dot is needed even if you never want to look at the dot files directly. Dot can be installed on windoze by merely unpacking the zip and adjusting PATH.

Some sybase installations have internationalisation set up in such a way that the URL needs to end in ?charset=iso_1. schemaSpy forms the URL based on a config it reads so the config must be extracted from the schemaSpy jar (a zip).

I first came across the charset issue when setting up database connections in DbVisualizer. I found
this web page which has the solution. This is also the solution for the same problem in schemaSpy.

Edit a copy of this config, appending ?charset=iso_1 to the name of the URL, then invoke schemaSpy with this command line (the sybase config filename is sybase.properties):

Note that the schema dbo must be specified, otherwise it tries to use the username as the schema name.

java -jar schemaSpy_5.0.0.jar -t sybase.properties -db <dbName> -u <userid> -p  <password> 
  -o <outputDir> -host <stringFromSybaseInterfaceFile> -port <portNumber>
  -dp <pathnameTojconn3.jar> -s dbo

Saturday, February 01, 2014

a high-volume, low-latency market data processing system implemented with IBM middleware

Way back in 2003 I worked on a very interesting C++ CORBA project. It was to do with the near real-time distribution of market data to multiple subscribers. Most of the work was contracted out to that well known consultancy, Object Computing Inc. IMHO they did a very good job. One of the things that made it interesting was the way they used an XML configuration file to describe a pipeline of components through which the messages passed. This used the Configurator pattern described in POSA.

Back in January 2013 I came across a system that reminded me strongly of the one I worked on. I found an article in the Wiley Online Library (WOL). Unfortunately the WOL article is behind a paywall. Luckily, the authors also made it available in several other places including one where you get get it as a PDF.

This is a reminder to me that OCI aren't the only ones to develop a system like the one I saw and that there is very good write up about that other system. Here is what is says in the abstract:

A stock market data processing system that can handle high data volumes at low latencies is critical to market makers. Such systems play a critical role in algorithmic trading, risk analysis, market surveillance, and many other related areas. We show that such a system can be built with general-purpose middleware and run on commodity hardware. The middleware we use is IBM System S, which has been augmented with transport technology from IBM WebSphere MQ Low Latency Messaging. Using eight commodity x86 blades connected with Ethernet and Infiniband, this system can achieve 80 μsec average latency at 3 times the February 2008 options market data rate and 206 μsec average latency at 15 times the February 2008 rate.

Sunday, December 15, 2013

Eclipse, CDT and the codan syntax checker

I have been using eclipse CDT recently on a legacy C++ project and it took a while for me to get the codan syntax checker to work. When it wasn't working it gave syntax errors for things that weren't syntax errors and claimed it couldn't find standard headers like stdlib.h, string.h etc. Use this command to invoke the GNU compiler so that it tells you where it is looking for includes:
g++ -E -x c++ - -v < /dev/null
These are the include directories that need to be configured. I got this tip from StackOverflow at http://stackoverflow.com/questions/11946294/dump-include-paths-from-g

You need to turn on full tracing when invoking eclipse (as explain my earlier post), then you will see if all the paths are resolved or not.

There are lots of posts of how to solve the codan problem but they are tend to be superficial and dismissive, often saying things like "put /usr/include" in the path. That is not good enough, you have to put in all the paths that the compiler uses, including the internal ones. Don't forget that when using cygwin and codan that the paths will have to be DOS-like, i.e. starting with the C:/cygwinDir..blahBlah.

Monday, December 02, 2013

How to install eclipse plugins on a machine that is cut off from the internet

Recently I have been using a proprietary eclipse plugin in a corporate environment. The plugin seems to assume java and windoze but the project was in unix-specific C/C++ and on linux. So I was faced with the task of setting up the plugin on linux. It was then I discovered that all the linux machines are cut off from the internet. What to do? I asked around and I amazed to discover that apparantly, no-one else has ever had to deal with this issue. Lots of people guessed that you just copy the jar files over from the plugins directory and restart eclipse. Wrong.

Copying files from the plugin directory plays a part in the solution but is by no means the be all and end all. For a start, sometimes the plugin is OS-specific, so it might contain DLLs (windoze) or shared libraries (UNIX). Any such plugin has to be obtained on the target machine somehow.

One of the main challenges of installing your own plugins without using a network connection is that you will have to chase down any dependencies. By default, eclipse does not tell you when there is a problem installing plugins but you really do need this information. Here is what you do:

create a file called .options which contains:
org.eclipse.equinox.p2.core/debug=true
.org.eclipse.equinox.p2.core/reconciler=true

then enter this command:
./eclipse -clean -console -consoleDebug -debug ./.options

This produces output that tells you what troubles it had loading plugins.

Under the eclipse directory create a directory called dropins. It should be a peer of plugins and features. Below dropins create eclipse and below that create plugins and features.

For any plugins that are missing, copy them from a windows environment that has them. They can be directories, files or a combination of the two. Be careful to copy to the correct target directory, either dropins/eclipse/features or dropins/eclipse/plugins depending on where it comes from in the source. To copy them, using WinCp. This is often allowed in a corporate environment.

Monday, October 07, 2013

C++ bollocks

Every now and then I come across what I call "C++ bollocks". What do I mean? I am referring to various statements made about certain aspects of C++ that are, well, complete bollocks. The statements might have been true to some extent in the
dim and distant past, but anyone still trotting out these statements in this day and age is, in my arrogant opinion, talking bollocks. Here are some examples, divided into categories.

all too frequent bollocks

  • C++ iostream are too slow. Use the printf family instead.
    see http://www.parashift.com/c++-faq/iostream-vs-stdio.html
  • C++ exceptions are too slow. Use error codes or boolean return status instead.
    see http://www.parashift.com/c++-faq/exceptions-avoid-spreading-out-error-logic.html. Also see http://www.stroustrup.com/bs_faq2.html, "Why use exceptions?"
  • Don't use std::endl, it's too slow.
    It is true that std::endl flushes the output buffer, which has a performance hit. This is even mentioned in the FAQ-lite at http://www.parashift.com/c++-faq/endl-vs-slash-n.html However, such concerns over micro-efficiency are usually misplaced. Premature optimisation is the root of all evil.
  • C++ strings are too slow. Prefer C char arrays.
    Some people prefer C to C++. Those people should leave C++ alone and work in their preferred language. Also see http://www.stroustrup.com/bs_faq2.html, "What's wrong with arrays?".
  • avoid virtual functions. The performance hit on virtual dispatch is too great.
    See http://www.parashift.com/c++-faq/dyn-binding2.html
  • always implement getters and setters as inline. It's faster.
    see http://www.parashift.com/c++-faq/inline-and-perf.html. As ever, premature optimisation is the root of all evil. People making these claims never have the figures to back them up. Their response to being challenged is "everyone knows this".
  • provide convenient implicit type conversion by implementing cast operators
    Actually you usually want to do the complete opposite of this. This is why many coding guidelines say that single arg ctors must be qualified by explicit. and that cast operators should not be written.
  • Failing to make a base class dtor virtual is no big deal. It will just result in a small memory leak.
    Wrong. It is actually undefined behaviour. See http://www.parashift.com/c++-faq/virtual-dtors.html
    and http://www.catb.org/jargon/html/N/nasal-demons.html
  • saying delete mem instead of delete []mem (to go with the new type[count]) is no big deal. It will just result in a small memory leak.
    Wrong. It is actually undefined behaviour. See http://www.parashift.com/c++-faq-lite/delete-array.html.
    and of course, http://www.catb.org/jargon/html/N/nasal-demons.html

infrequent bollocks

  • don't throw exceptions from constructors.
    this is from the same school of bollocks as "C++ exceptions are too slow".
    See http://www.parashift.com/c++-faq/ctors-can-throw2.html
  • prefer pointers to references since references dont make it clear when the object can be modified.
    see http://www.parashift.com/c++-faq/refs-vs-ptrs.html and
    http://www.parashift.com/c++-faq/ref-semantics-sometimes-good.html


Thursday, September 26, 2013

A nasty .Net error in Visual Studio 2010 and how I fixed it

Every now and then I got a weird error whilst tinkering with some Visual Studio project files. The error was like this:

22>------ Build started: Project: myLibraryDll, Configuration: Release x64 ------
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(298,5): warning MSB8004: Intermediate Directory does not end with a trailing slash.  This build instance will add the slash as it is required to allow proper evaluation of the Intermediate Directory.
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018: The "VCMessage" task failed unexpectedly.
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018: System.FormatException: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at System.Text.StringBuilder.AppendFormat(IFormatProvider provider, String format, Object[] args)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at System.String.Format(IFormatProvider provider, String format, Object[] args)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.Shared.ResourceUtilities.FormatString(String unformatted, Object[] args)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.Utilities.TaskLoggingHelper.FormatString(String unformatted, Object[] args)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.Utilities.TaskLoggingHelper.FormatResourceString(String resourceName, Object[] args)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.Utilities.TaskLoggingHelper.LogWarningWithCodeFromResources(String messageResourceName, Object[] messageArgs)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.CPPTasks.VCMessage.Execute()
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask, Boolean& taskResult)
This is quite obscure, I think you'll agree. I eventually tracked it down to a spurious trailing slash on OutputFile. Remove it and the error goes away.

I have been bitten by this several times now so I thought if I blogged about it, then I would remember.