Last Friday I was about to leave from office but wanted to deploy a new version of Neox-bot with some exciting features. I was initially working on the new features and when I was doing that, I wanted to make some changes to the MongoDB configuration on the Ubuntu server using SSH connection and restart MongoDB. That is when I ended up with a strange error message and it almost drained my energy last Friday. I wanted to share that experience as an article so that people who might have similar issues can get some clue from the article.
My intention was to make some changes to the MongoDB configuration to make sure the database server is public and is accessible from anywhere. In order to do that I had to make some changes to the config file. After making changes, I read online that if I make changes to the mongod.conf file, I will have to restart the server. So I wanted to restart MongoDB services first and if things don’t work out well, I was planning to restart the server itself. I used the below command to restart MongoDB service.
sudo service mongod restart
After restarting the service, I wanted to check the status of the service and for that I used the below command.
sudo service mongod status
That is when I ended up with lots of error messages. First of all the MongoDB service was working fine before I restarted it and after the restart, it suddenly stopped working. So, I was under the impression that the issue was because of the new changes that I made to the config file though it did not make any sense to me, I did rollback the changes I made to the config files. Initially, the error messages that were displayed there were inclined more towards issues in the configuration files. I was solving them one by one.
It did give me a clue that the issue was related to authorization. I changed the access of the user and tried using root access to restart the service and I was still getting the same error message. Then later I realized that the files should have been in different user access and checked them using the below command.
This command also confirmed that all the files were having permissions for root and the access rights were also proper. It was still a puzzle for me. With no other options available, I thought of reinstalling MongoDB with a proper user account instead of installing it with a root account. Some contents online suggested that it is not ideal to use the root account to install and manage applications because other users who log in to the server can have issues.
Before trying to reinstall MongoDB, I wanted to take a backup of the existing databases in the server so that I can go ahead with a clean slate re-installation. I completed taking a backup of the existing databases and wanted to check a few more options. I took some time to just relax and first understand the issue. I was trying to restart mongod service and that gave me an error message. That is when something hit me really hard in my mind. I was curious to go and check the mongod service config file to see what user account access the file is using. It was mentioned as mongodb user which was not even there in the server.
I was finally relaxed that it should have been the root cause of the issue and I changed the user account in the config file to root user and restarted the mongodb service. Yes, it was good news for me that the service started without any issue. It was an exhausting 4 hours exploration for the RCA. But the good thing about this issue is that it gave me lots of knowledge and I was able to learn some new things while trying to troubleshoot the issue. The only concern for me is that all this happened on a Friday night.