Skip to content

Troubleshooting Guide

Web Interface Errors and Probable Issues

1. The error occurs in the web interface/Case 1

Error: An Internal Error Has Occurred. Please check that the required services are running.

Probably the mysql server is off/crash. Check with ssh with

systemctl status mysql.service

If the service does not look like it's running:

mysql.service - LSB: Start and stop the mysql database server daemon
 Loaded: loaded (/etc/init.d/mysql)
 Active: inactive (dead) since Mon 2016-09-12 09:37:28 EEST; 1min 43s ago
 Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
 Process: 548 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)

The problem is resolved with the mysql.service restart system

Check with systemctl status mysql.service

mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: active (running) since Mon 2016-09-12 09:40:48 EEST; 2s ago
  Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
  Process: 15959 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)

2. The error occurs in the web interface/Case 2

Error: Connecting to the main datastore. Are all the services running? Connection refused

Probably the mysql server off/crash. Check with ssh with systemctl status mysql.service

If the service does not look like it's running :

mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: inactive (dead) since Mon 2016-09-12 09:37:28 EEST; 1min 43s ago
  Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
  Process: 548 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)

The problem is resolved with the systemctl restart mysql.service*

Check with systemctl status mysql.service

mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: active (running) since Mon 2016-09-12 09:40:48 EEST; 2s ago
  Process: 15510 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
  Process: 15959 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCES

OR

Probably server elasticsearch is off/crash. Check with ssh with systemctl status elasticsearch.service

If the service does not look like it's running :

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
   Active: inactive (dead) since Mon 2016-09-12 09:41:48 EEST; 2min 5s ago
     Docs: http://www.elastic.co
  Process: 608 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=143)
  Process: 517 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 608 (code=exited, status=143)

The problem resolves with systemctl restart elasticsearch.service

Check with systemctl status elasticsearch.service

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
   Active: active (running) since Mon 2016-09-12 09:44:28 EEST; 2s ago
     Docs: http://www.elastic.co
  Process: 16854 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 16856 (java)
   CGroup: /system.slice/elasticsearch.service
           └─16856 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX...

3. The error occurs in the web interface/Case 3

Error: Connecting to the main datastore. Are all the services running? ElasticSearch Error: Error

Probably mysql server off/crash. Check with ssh with mysql.service systemctl status

If the service does not look like it's running:

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
   Active: inactive (dead) since Mon 2016-09-12 09:41:48 EEST; 2min 5s ago
     Docs: http://www.elastic.co
  Process: 608 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=143)
  Process: 517 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 608 (code=exited, status=143)

The problem resolves with systemctl restart elasticsearch.service

Check with systemctl status elasticsearch.service

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
   Active: active (running) since Mon 2016-09-12 09:44:28 EEST; 2s ago
     Docs: http://www.elastic.co
  Process: 16854 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 16856 (java)
   CGroup: /system.slice/elasticsearch.service
           └─16856 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX...

4. Mysql service off

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status mysql.service

Alt text

and the error following commands:

Alt text

now if we go into the web application this will be the error that will occur:

An Internal Error Has Occurred. Please check the that required services are running.

Alt text

The resolving method is to restart the service by following the command:

systemctl restart mysql.service

Alt text

and the result of the command is the following:

systemctl status mysql.service

Alt text

5. Elasticsearch service off

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status elasticsearch.service

Alt text

and the error following commands:

Alt text

now if we go into cerebro web plugin (http://serversi:9000) will show the following error

Alt text

or if we go into the web application this will be the error that will occur:

There is no data available

Alt text

The resolving method is to restart the service by following the command:

systemctl start elasticsearch.service

Alt text

and the result of the command is the following:

systemctl status elasticsearch.service

Alt text

6. Rabbitmq-server service stopped

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status rabbitmq-server.service

Alt text

and the error following commands:

Alt text

now if we go into rabbitmq web plugin (http://servers:15672) does not work

Alt text

The resolving method is to restart the service by following the commands:

Alt text

and

Alt text

and

Alt text

and the result of the command is the following:

systemctl status rabbitmq-server.service

Alt text

7. Nginx.service service stopped

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status nginx.service

Alt text

and the error following commands:

Alt text

or if we go into the web application this will be the error that will occur:

Alt text

The resolving method is to restart the service by following the command:

systemctl restart nginx.service

Alt text

and the result of the command is the following:

systemctl status rabbitmq-server.service

Alt text

8. Php5-fpm.service service stopped

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status php5-fpm.service

Alt text

and the error following commands:

Alt text

or if we go into the web application this will be the error that will occur:

Alt text

The resolving method is to restart the service by following the command:

systemctl restart php5-fpm.service

Alt text

and the result of the command is the following:

systemctl status php5-fpm.service

Alt text

9. Data-storage.service service stopped

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status data-storage.service

Alt text

and the error following commands:

Alt text

events blocked on the queue in Data Storage on RabbitMQ (http://servers:15672)

Alt text

The resolving method is to restart the service by following the command:

systemctl restart data-storage.service

Alt text

and the result of the command is the following:

systemctl status data-storage.service

Alt text

10. Data-acquisition.service service stopped

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

systemctl status data-acquisition.service

Alt text

and the error following commands:

Alt text

events blocked on the queue in data-acquisition.service on RabbitMQ (http://servers:15672)

Alt text

The resolving method is to restart the service by following the command:

systemctl restart data-acquisition.service

Alt text

and the result of the command is the following

systemctl status data-acquisition.service

Alt text

or if we go into the web application this will be the error that will occur:

Connecting to the main datastore. Are all the services running? Index “el_logs_current” is missing

Alt text

now if we go into cerebro web plugin (http://serversi:9000) will show the following error:

Alt text

The resolving method is to restart the service by following the command:

systemctl restart data-acquisition.service

Alt text

11. Rsyslog service stopped (self-audit events)

The error which appears in the “putty” with the corresponding host name (or ip address) and the port and also the connection type (SSH) using the command:

/etc/init.d/rsyslog status

Alt text

and the error following commands:

Alt text

The resolving method is to restart the service by following the command:

/etc/init.d/rsyslog restart

Alt text

and the result of the command is the following

/etc/init.d/rsyslog status

Alt text

Or another problem can be found in ( nano /etc/rsyslog.conf ):

Alt text

Scroll down until you reach the end of the file and check if it matches the desired ip (" *. * " - this symbol represents all types of events)

Alt text

12. RabbitMQ error

When this error occurs, we don’t find new information in the web application. We find this error in CyberQuest logs. Using the baretail program we open the file(agent.txt) and we can view the error log.

Alt text

To solve the problem we have to stop the Cyberquest Agent and restart the RabbitMQ service. Follow the instructions for more details:

A. Stop CyberQuest Agent

On the Windows machine where the CyberQuest Agent is installed, open Windows Services and stop the CyberQuestAgent service.

Alt text

To check if the CyberQuest Agent has stopped, open Task Manager > Details and wait until the Agent .exe process disappears from Task Manager.

Alt text

B. Restart RabbitMQ

We connect to the CyberQuest server via ssh (e.g. ssh user@ip) and restart RabbitMQ using the following command:

systemctl restart rabbitmq-server.service

C. Start CyberQuest Agent

On the Windows machine where the CyberQuest Agent is installed, open Windows Services and start the CyberQuestAgent service.

Alt text

To verify that the CyberQuest Agent has started, open Task Manager> Details and wait until the Agent.exe process appears in the Task Manager.