Week 2

Day 1

Today I was mainly focused with setting up my CMakeLists.txt file and building my plugin. I began today reading over last weeks blog and it was a helpful refresher for what I was working on last week, especially with the long weekend that we had. Once I got back in the right frame of mind I was ready to start building my CMake file. I first looked at the couchbase plugin a few more times and was able to understand a few more things than when I looked over it last week. I think repetition for me has been my biggest tool for understanding the new system, and I think I am starting to become acquainted with it. I finished pulling most of everything I needed from the couchbase plugin just in time for my daily catchup meeting with Dan.

The meetings with Dan have been very helpful and today he walked through some of the code from the CMakeLists.txt file from the kafka plugin. There was some automation in there that will be similar to what I need to do, but for now since the driver is already set up I don't need to worry about coding that just yet. He gave me a few more pointers that taught me some things about CMake and since then I have been looking over CMake's documentation.

I finally started doing some actual coding which felt great because up until now I have just been mainly getting everything set up, so it was good to finally make some progress and to have something to show for it. There were two files that I needed to edit in the HPCC systems CMake files that would tell it to look for my plugin and to try to build it. The top level CMakeLists.txt file had a funtion called HPCC_ADD_SUBDIRECTORY which had two arguments I needed to pass to it. I had to tell it the directory where my plugin was to be built, plugins/mongodb, and the name of the plugin found in the CMakeLists.txt file which is MONGODBEMBED. In another file located in the cmake_modules directory called commonSetup.cmake there was a list of plugins where I had to add MONGODBEMBED to it. I had first tried building my plugin without making a change to the second file, and that resulted in an error where it didn't try to build it at all and just built the rest of the system.

Once I fixed that error and tried building it again I was finally able to get some error messages that allowed me to start debugging my code. This was the first time in the project where I was actualy able to write some code, test it, and try to fix the error, so now I feel much better about the progress that I am making and I feel like I am not so lost anymore which is great. I am looking forward to tomorrow where I can keep debugging and hopefully get the plugin built with all of its dependencies without having any errors.

Day 2

I began today working on sorting out the errors that I encountered when trying to build my plugin. In the C++ driver for MongoDB that I installed last week there was a directory called mongocxx that contained a lot of header files that are required to use the driver. In some cases the header files are built which is what I found out later after doing some debugging. Because some of these files are built when you call cmake there was an instance of duplicate directories and my CMakeLists.txt file was including the wrong one as a subdirectory. Once I figured that out it seemed to fix it as it threw a different error saying a different file was missing. This time I needed to include some files from bsonxx. Bsonxx is the data type that MongoDB works with for the most part. It is similar to JSON datatypes but provides a lot of methods for building documents and converting types and etc.

After working through a few of the bugs and having to change the CMakeLists.txt file a lot of times I realized working from the terminal was not going to be very efficient. I know what you're going to say, "You were working from the terminal this whole time? What text editor were you using? How annoying was that?" Yes, I was using nano until now and having to constantly go in and out and change directories was pretty cumbersome. I am pretty comfortable working from the command line so it was not difficult for me but there are much better options. With the level of development I will be doing it was necessary to come up with a better solution for editing the files and looking through directories. I decided to set up Visual Studio Code to edit the files remotely and this was the first time I had used it for that. I had done something similar with netbeans in one of my classes, but since it had been such a long time I decided to just use Visual Studio Code since I already had it installed and I had used some of the other features that it offers much more recently.

To get it set up I had to install OpenSSH and that was pretty simple I just ran some commands in the terminal and it was pretty easy.

To download OpenSSH I used this command:

Then to install the tools that it downloaded I ran these commands:

From there the only thing I needed to do was install this extension for VS Code and the easy part was done. The next part was a little tricky, but I am glad I chose to do it this way as I learned some valuable lessons for working with SSH keys and remote options for development. The first step, which I actually did like 5th because I was learning on the go, was to convert the SSH key that I used for putty into a file that windows could use. I used PuTTYgen for this and tool is very simple and easy to use. I just selected convert and chose SSH key and saved it in a ~/.ssh. Then the thing that I actually did first before I had the right key to use was go into the ssh config file at ~/.ssh/config and added a profile for the server so that VS Code would be able to connect to it properly. If you don't configure the config file correctly you'll get the error "Permission denied (publickey)." That was frustrating how many times I got that error, but I knew it was just because it was not using the SSH key correctly and once I got that fixed I was able to connect to the VM via VS Code and see all the directories much more easily than having to cd and ls all the time.

Getting VS Code set up properly and being able to have all the directories right next to the code and not have to exit out of it all the time and jump back in really made such a huge difference and I am glad I finally got that setup. I also have a much better understanding of what is going on with the .ssh/config file and how that works which is a great takeaway from all this. Now my work will be much more focused and I was able to get back to fixing link errors in my CMakeLists.txt file which I am happy about. Tomorrow my goal will be to get my plugin to successfuly build, but I cannot tell how many errors I have made until I fix the last one so it might take me a lot longer or I may only have one left I will have to keep on going for now.

Day 3

Today I was back at it with fixing link errors and making sure my cpp file was finding all of the dependencies that it needed to. After a few more changes I was able to fix all the errors that had to do with missing files and started getting into fixing basic errors in my code. There were some instances where I was using an object that hadn't been implemented yet, so I had to comment a few lines out and some functions that I wouldn't need until later. I then had to copy some more code over from the couchbase plugin because I hadn't defined some of the methods that I was using in the cpp file in the header file, and once I added all the code I was missing it finally built. It was a good feeling to get to 100% and see that the build was completed after nearly two days of trying to get it working. It was slow and still is, but my progress feels like it is accelerating. I am learning a lot and it feels like I am reaching the difficult part of the learning curve as my proplems change and get more difficult. Luckily, I have Dan who has been helping me with things that I have never seen before and it doesn't feel like I am too stagnant.

Once the build was complete I had to start making it into a package by calling this command after cmake:

	make -j3 package

The -j3 flag tells the scheduler how many threads to use when creating the package. The HPCC Systems recommends using 50% more threads than are currently available, but I have tried -j6 and -j3 and it does not seem like there is a noticeable difference so I just decided to use -j3. I think this is due to the size of my package being very small right now and so the difference might be larger once the package starts to get to full size. Once the package was made I tried to install it with this command:

sudo dpkg -i ./hpccsystems-plugin-mongodbembed_<version name>.deb

This threw an error at first saying that there were mismatched versions. The way that I fixed it was to rebuild the HPCC-Platform in the other build directory and use the make command on that. This way the versions would be automatically synced as they would come from the same source. Once that was done I was able to install the MongoDB package fine, but when I went to verify that everything was installed correctly I didn't find the necessary files that would have been created if everything worked. HPCC-Platform installs into the top level /opt /HPCC-Platform directory. With in that there is a plugins folder that holds the .ecllib files and other important files that should be there. There wasn't anything from MongoDB in there which was concerning because that means my CMakeLists.txt file was incorrect.

In order to get an idea of what was supposed to be in the /opt/HPCC-Platform/plugins directory other than the .ecllib folders I built the kafka plugin and installed that as a package. When I was doing this Dan told me that it was really important to build everything with the -DCMAKE_BUILD_TYPE=Debug flag because this would give me necessary debug information for everything I am trying to do. He also pointed me to a StackOverflow post that had a similar issue that I was having and in the solution they showed me a directory of example mongocxx driver applications that used CMake and I started looking through that to try and fix my CMakeLists.txt file. I unfortunately ran out of time today before I could figure out what I was missing or what was broken about my CMakeLists.txt file, but now that I have a lot more information about the problem with the Kafka plugin example, mongocxx driver examples, and the knowledge that the issue lies with my CMakeLists.txt file I feel prepared to solve this problem and that will be my goal for tomorrow.

Day 4

Today my goal was to figure out why the install was not working correctly, but before I could do this I needed to try a few simple fixes to see if I understood the problem. The problem was that it was not putting the necessary files in the proper location. To get an idea of what kind of files I needed to see there I used the build of the kafka plugin and ran this command:

dpkg-deb -c ./hpccsystems-plugin-kafka_<version name>.deb

The output that I got from the command (with a column removed to fit better) was:

drwxrwxr-x root/root ./opt/
drwxrwxr-x root/root ./opt/HPCCSystems/
drwxrwxr-x root/root ./opt/HPCCSystems/lib/
lrwxrwxrwx root/root ./opt/HPCCSystems/lib/librdkafka++.so -> librdkafka++.so.1
-rwxr-xr-x root/root ./opt/HPCCSystems/lib/librdkafka++.so.1
lrwxrwxrwx root/root ./opt/HPCCSystems/lib/librdkafka.so -> librdkafka.so.1
-rwxr-xr-x root/root ./opt/HPCCSystems/lib/librdkafka.so.1
drwxrwxr-x root/root ./opt/HPCCSystems/plugins/
-rw-r--r-- root/root ./opt/HPCCSystems/plugins/libkafka.so

When I ran the above command on the .deb file for my plugin I got this output:

drwxrwxr-x root/root         0 2022-06-03 19:49 ./opt/
drwxrwxr-x root/root         0 2022-06-03 19:49 ./opt/HPCCSystems/

So clearly, I was pretty much missing everything. I needed to find out why I was missing those files and for that I turned to the mongodb github where I found some example files for using cmake to build an application similar to what I was doing. Unfortunatley, they were doing it a different way than I would like. They were using find_package to link all the packages and I wanted to do that manually to avoid adding additional requirements for the user to use the plugin. At this point I was a little stumped and not sure where to go.

Immediately after my morning chat with Dan I was able to identify part of the problem after he gave me some clarifying information. I had been wondering about the .so file extension and what it was exacty, so I asked Dan about and it is essentialy the shared library files which are used at runtime. In hindsight, I should have asked him about those a few days ago, but here we are learning about it today. There were three different types of .so files and they essentially all link together to provide the full shared library. This was useful information because previously I had only told CMake about one of the files that it needed and I had never installed it using the install command in the CMakeLists.txt file. Once I added the install commands and pointed CMake in the direction of the other two .so files it all seemed to work fine until I noticed that only the mongocxx files had made it into /opt/HPCCSystems/plugins. It was missing all the files for bsoncxx which was a little concerning. I went through the code again and was looking for any differences in the mongocxx part and the bsoncxx part. Originaly I thought it might have something to do with me not including the underlying C drivers that the C++ drivers are built on but after adding those dependencies I still wasn't getting the bsonxx files to show up. Now, it took me longer than I would like to admit but eventually I found the typo. In some places I had put bisoncxx instead of bsoncxx because when I say it in my head I pronouce it bison but really it is just a variation of JSON datatypes hence the BSON. Once I discovered that and replaced all the instances of bisoncxx with bsoncxx all the correct files showed up.

Fortunately, I was able to make some decent progress today and this week, but it is still not quite finished. The plugins file was missing the mongodb.ecllib file. I haven't been able to figure out why yet as the file was there for the kafka plugin even though the package was never installed. This will be my goal for next week and once I get that file added correctly I will be in a good place with my plugin and can hopefully start trying to get it to connect to MongoDB servers.