Software engineering, problem solving, nerd stuff

Fixing “Unable to obtain lock on store lock file” when using the embedded database of Neo4j

After some years without using Neo4j I had the chance to use the graph database in my current project. Neo4j was a good fit because it makes it really easy to prototype an idea and the project itself relies heavily upon tree structures. Modelling tree structures in relational database management systems is always a PITA and takes a certain time to implement in the backend. The good news is that trees are only directed graphs which can be easily modelled with Neo4j.

As a Spring guy I used the latest spring-data-neo4j 4.1.x release to connect the application to the graph database. For easier prototyping I used the embedded driver.

import org.neo4j.ogm.session.SessionFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.neo4j.repository.config.EnableNeo4jRepositories;
import org.springframework.transaction.annotation.EnableTransactionManagement;

@EnableTransactionManagement
@EnableNeo4jRepositories(basePackages = "de.schakko.neo4j")
@Configuration
public class Neo4jConfiguration extends org.springframework.data.neo4j.config.Neo4jConfiguration {
	@Override
	public SessionFactory getSessionFactory() {
		return new SessionFactory(getConfiguration(), "de.schakko.neo4j");
	}
	
	@Bean
	public org.neo4j.ogm.config.Configuration getConfiguration() {
		org.neo4j.ogm.config.Configuration config = new org.neo4j.ogm.config.Configuration();
		
		config.driverConfiguration()
		.setDriverClassName("org.neo4j.ogm.drivers.embedded.driver.EmbeddedDriver")
		.setURI("file:///var/tmp/graph.db");
		
		return config;
	}
}

When running the integration tests (or after automatic reloading of the application by spring-boot-devtools) the application failed to reconnect to the database:

org.neo4j.kernel.StoreLockException: Unable to obtain lock on store lock file: file:///var/tmp/graph.db

In the end I fixed the error by using

import org.junit.After;
import org.junit.runner.RunWith;
import org.neo4j.ogm.drivers.embedded.driver.EmbeddedDriver;
import org.neo4j.ogm.service.Components;

class ItemRepositoryIT {
	@After
	public void tearDown() {
		((EmbeddedDriver) Components.driver()).close();
	}
// ...
}

in the integration tests.

Open Tabs: Resources for Web Developers, How to write perfect blog posts, Argus

Resources for Web Developers

  • Freepik is a search engine for finding free vector graphics, PSDs, icons and and photos.
  • If you have to quickly create a landing page for your AppStore product, you can use the free service Landing Harbor.
  • Searching for a tool for your freelancer job? Take a look at The Freelance Stack.
  • Need a new Bootstrap-based theme? Bootstrap Zero should give you a good start.

How to write better blog posts

You have no idea how to start your next blog post? Try the free Content Idea Generator. And after you have found your topic you should definitely follow the rules of the awesome article After 10,000+ data points, we figured out how to write a perfect Medium post. The article contains a lot of hints how to optimize your blog posts.

Monitor your time series with Argus

Argus is an open source project proided by Salesforce to monitor your time series and receive alerts. I will take a deeper look at this project in the coming weeks.

lumiverse.io

On lumiverse.io you can find some interesting videos about neural networks.

Making money with GitHub

I stumbled upon these two projects on HN: CodeMill and Bountysource both provides a marketplace service for GitHub pull requests. You can search for open tickets, make a pull request and get paid by the community or the project owner.

GitHub project management

zube.io is a payed service to organize your GitHub projects. The open source alternative gh-board can be hosted on your own servers and has a lot of features like linking multiple GH repositories to one Kanban board.

Migrating InfluxDB from 0.9.6 to 0.10.0 GA

You may have already heard that InfluxDB 0.10 GA has been published a few days ago. In my case the most interesting improvement are the much higher compression rates: At the moment we are collecting performance data from four internal virtual machines. Have been running the collectd daemon on the VMs for about a month, the size of InfluxDB’s data directory increased by 3.5 GByte with default retention policy.

Testing the update

After setting up a new virtual machine with InfluxDB 0.9.6 for migration testing, I updated InfluxDB to the current version (Fedora 23 in our case):

sudo wget https://s3.amazonaws.com/influxdb/influxdb-0.10.0-1.x86_64.rpm
sudo dnf install influxdb-0.10.0-1.x86_64.rpm

Two notes:

  1. As mentioned during the update procedure, the location of the configuration file moved from /etc/opt/influxdb/influxdb.conf to /etc/influxdb/influxdb.conf. The old configuration file is neither copied nor merged. You must edit the new configuration file or copy the old configuration to the new location. Keep in mind that 0.10.0 introduces some new settings.
  2. In addition to the changed configuration location the data directory has been moved from /var/opt/influxdb/ to /var/lib/influxdb. My advice is to move the old folder to the new location and overwriting any new files. Please create a backup from the /var/lib/influxdb folder before doing this. In my case this procedure worked without any problems.

Unfortunately it doesn’t seem to be possible to just copy a single database from one separate InfluxDB instance to another. Because of this drawback I imported some data from our virtual machines through Logstash and InfluxDB’s collectd backend. The Grafana dashboards still worked as expected so there seems to be no breaking changes in the HTTP API, at least as far as I could see.

Converting data from bz1 to tsm1

Now it was time to convert the data from the old bz1 format to the new tsm1. InfluxDB 0.10 delivers a nifty tool named influx_tsm. The usage is very easy and it worked as expected. Just pass the parameter “-backup” or “-nobackup”, the data location to be converted and you are done:

systemctl stop influxdb
influx_tsm -backup ~/inflxudb_backup -parallel /var/lib/influxdb/data/
systemctl stat influxdb

Results of the conversion

In my test environment everything worked as expected so I migrated our production environment. The influx_tsm took around ~50 minutes for 3.6 GByte of data but the results are awesome. Our data directory shrinked from 3.6 GByte to 400 MByte. All Grafana dashboards work as expected as well as our collectd and Graphite InfluxDB backends. This is the output of influx_tsm:

# ...
2016/02/07 13:34:29.100186 Still Working: Completed Shards: 29/30 Points read/written: 81626859/81626859
2016/02/07 13:34:31.969792 Conversion of /var/lib/influxdb/data/collectd/default/60 successful (49m41.016488953s)

Summary statistics
========================================
Databases converted: 3
Shards converted: 30
TSM files created: 30
Points read: 81655978
Points written: 81655978
NaN filtered: 0
Inf filtered: 0
Points without fields filtered: 0
Disk usage pre-conversion (bytes): 5239898112
Disk usage post-conversion (bytes): 451150534
Reduction factor: 91%
Bytes per TSM point: 5.53
Total conversion time: 49m48.08131854s

Thanks guys, good job!

Executing Liquibase database migrations from command line and as a shared Maven JAR

I am currently working on the migration of our time tracking system from Microsoft SQL Server/.NET to Java. Most of the logic resides in Stored Procedures and Stored Functions inside the database schema. Because of some reasons (testability, maintainability, migration from MSSQL to PostgreSQL in a far future) the whole logic must be converted to Java. As the system has a high criticality all of the end user applications must be running parallel. There are 4 tools in total, written in different languages: an old PHP web application, a bridge from JIRA to our time tracking system, another JIRA-to-time-tracker converter and the original C#/.NET fat client. All systems will be migrated bit by bit to the new Spring Boot web application.

Using Liquibase for database versioning

After collecting information about the current application environment I noticed that there were no database versioning system in use. The installation of the MSSQL schema was a pain: there were a lot of plain SQL files which had to be executed by hand. Since Java was the target programming language I decided to use Liquibase and moved the whole SQL scripts into a new Git repository, added the pom.xml and wrote a self-explaining Readme.md how the .NET developers had to use Liquibase.

Running Liquibase migration with the command line Maven plug-in

I decided to describe only the pure Maven approach and not the Liquibase installation. The execution of the Liquibase migrations are trivial and no dependencies had to be installed by hand. The pom.xml contained the following definitions:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>my.package</groupId>
	<artifactId>database-schema</artifactId>
	<version>1.0.0</version>
	<packaging>jar</packaging>
	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	</properties>
	<dependencies>
		<dependency>
			<!--  from internal Artifactory; Microsoft does not make the sqljdbc4.jar available in official repositories -->
			<groupId>com.microsoft.sqlserver</groupId>
			<artifactId>sqljdbc4</artifactId>
			<version>4.0</version>
		</dependency>
	</dependencies>
	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-resources-plugin</artifactId>
				<version>2.7</version>
				<configuration>
					<encoding>UTF-8</encoding>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.liquibase</groupId>
				<artifactId>liquibase-maven-plugin</artifactId>
				<version>3.0.5</version>
				<configuration>
					<propertyFile>src/main/resources/liquibase/liquibase.properties</propertyFile>
				</configuration>
				<executions>
					<execution>
						<goals>
							<goal>update</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
</project>

The liquibase.properties file did only contain default values and is not important. With the configuration above I were able migrate the MS SQL schema with

mvn -Dliquibase.url=jdbc:sqlserver://$HOST:1433;databaseName=$DATABASE-Dliquibase.username=$USERNAME -Dliquibase.password=$PASSWORD liquibase:update

At this point I could update the database schema by hand. This was necessary when someone had to develop inside a .NET environment or we had to migrate a staging or production database.

Making the schema available for developer environments

I realized quickly that the approach did not work really well for my Java development environment. A lot of database migrations had to be developed and the integration test environments should be automatically in sync with the defined migrations. My idea was to let Jenkins push the Liquibase defined schema as a Maven JAR into our internal Artifactory. I should be able to include the JAR as a normal Maven dependency and let Spring’s Liquibase integration to execute the latest migrations.

Reference a db-changelog.xml inside a JAR in your application.properties

I took a look in LiquibaseProperties and saw that the changeLog attribute supports the resource syntax. All I had to do was defining the db-changelog.xml by adding the following line to the application.properties:

liquibase.change-log=classpath:/liquibase/db-changelog.my-app.xml

Please note that I changed the filename from db-changelog.xml to db-changelog.my-app.xml. This should prevent ordering issues if there is already another XML file present with the same file name. The classpath prefix is used by Spring to scan all JARs in the classpath for the requested path.

Do not use the full path name of included SQL files

As I mentioned above all SQL statements resided in their corresponding SQL files. I used the following definition in the db-changelog.my-app.xml to include the SQL files:

	<changeSet id="5" author="ckl">
		<comment>Bla</comment>
		<sqlFile dbms="mssql" encoding="utf8"
			path="install/20151126-005_sp_calculate_worklogs.sql"
			relativeToChangelogFile="true"
			splitStatements="true" />
	</changeSet>

This worked if Liquibase was either executed only through the Maven command line or as a Maven JAR dependency, but not both.

How Liquibase calculate database migration differentials

Liquibase iterates through all changesets defined in your db-changelog.xml. The XML attributes id, author and path are used at first to check whether this migration already exists in the database table DATABASECHANGELOG. If a row with the given parameters does exist, the checksum of the SQL file is calculated by normalizing the SQL file content (replacing new lines and so on). After that a MD5 checksum is generated by using the header and the content of the file.

The content of the “path” attribute differs

When executing mvn … liquibase:update inside the Git repository, the column path is filled with src/main/resources/liqiuibase/install/20151126-005_sp_calculate_worklogs.sql. Executing the migrations during the Spring startup process will result in a value classpath:/liquibase/install/20151126-005_sp_calculate_worklogs.sql for the path columns.
This means that every migration will be executed again, resulting in DDL errors.

Ignoring the path attribute

The easiest way was to use the attribute logicalFilePath in my databaseChangeLog tag. This forces all rows to have the same value of the path column:

<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
	xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.0.xsd
    http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd"
	logicalFilePath="path-ignored">
....
</databaseChangeLog>

In the DATABASECHANGELOG table the column path is filled with path-ignored.

Do not mix up different Liquibase versions

After I had fixed the previous error Liquibase showed that the calculated checksum of the files differed. At first I thought I had some encoding issues and forced everything to UTF-8 but the error was still there. It took a while until I noticed that the Maven dependency liquibase-core in the Spring Boot app and the Maven Liquibase plugin for command line execution had different versions (3.3.2 versus 3.0.5). Both versions calculates the MD5 checksum in different ways. The checksum inside the DATABASECHANGELOG table differed with the newly calculated checksum. All I had to do was changing the Liquibase Maven plug-in to use the same version:

	<build>
		<plugins>
			<plugin>
				<groupId>org.liquibase</groupId>
				<artifactId>liquibase-maven-plugin</artifactId>
				<!-- same version -->
				<version>3.3.2</version>
				<configuration>
					<propertyFile>src/main/resources/liquibase/liquibase.properties</propertyFile>
				</configuration>
				<executions>
					<execution>
						<goals>
							<goal>update</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>

TL:DR

I moved the definition of a Microsoft SQL Server database schema into its own repository, made the schema migratable with help of Liquibase and made it executable in standalone/command line mode and as a Maven JAR dependency.

Open Tabs: Awesome CV, uxdesign.cc & Mobile Patterns, re:Work, d3.compose, cloudcraft.co, SqlPad

In the last week the following links stayed longer open in my browser tabs:

Awesome CV

If you are searching for a LaTeX template for your resume you should definitely take a look at Awesome CV. I will probably use the template in one of my projects.

uxdesign.cc and Mobile Patterns

As I already mentioned from time to time I am a lousy user interface designer. Nevertheless I am interested in UI and UX. uxdesign.cc has a ton of resources for user experience designers like tools, links and methods for concepting and planning interaction design.

Mobile Patterns is a platform for sharing screenshots of mobile applications. This site is awesome because you get a good insight how to create responsive web applications which are easy to use.

re:Work

re:Work describes steps to improve processes at your workplace.

Responsive d3.js charts with d3.compose

D3.js is awesome for creating interactive charts of every type. With d3.compose you can easily create responsive D3.js-based charts.

What I tell all new programmers

I noticed the blog post What I tell all new programmers on HN in the last week. I liked the content and linked the blog post in our internal wiki for apprenticeships.

README Driven Development

README Driven Development (RDD) means that you write the Readme.md before you start programming. The idea is not new but has been working for me since years, too. I use it for my personal projects in combination with something I like to call Sketched Driven Development (just kidding!): Sketching the entity relationships and workflows with my own simple form of UML.

Perceptual testing with VisualReview

Perceptual testing simply means comparing application screenshots of the baseline (= production environment) with screenshots of your staging environment. The screenshots can be captured with tools like Selenium or PhantomJS. Thoughtworks provides a good overview what perceptual testing means. Beneath the tools DPXDT, Viff and Pix-Diff mentioned in Thoughtwork’s blog post, the open source tool Visual Review from Xebia is worth a look.

cloudcraft.co – Visualize your cloud architecture like a pro

I don’t know how often I sweared about Microsoft Visio and other tools for modelling infrastructures. Cloudcraft.co could be a game changer in the long term. At the moment it “only” supports the visualization of AWS-based infrastructure but there is potential to use it for other infrastructures as well.

SqlPad

The Node.js based application SqlPad allows you to execute SQL queries on different DBMS’ like MySQL, PostgreSQL and Microsoft SQL Server and visualize the returned result set with different charts. The queries can be saved for later execution. The application is ideal for simple dashboards based upon different datasources.

Project idea: Nostradamus AKA prophetr – a social forecasting network for experts

From time to time some programming ideas come to my mind which I can not forget. I had often started a new project but due to my limited amount of free time it is hard to finish all of them. In the next blog posts I will describe my ideas with some technical and environmental background. If you are interested in getting more information, buying the idea/source code or just motivate me, just drop me a line at me[at]schakko[dot]de. I would really appreciate your feedback.

Making prophecies and verifying their occurence

It must been around five or six years ago when I first thought about the idea of making prophecies in technical topics. I remember that I had a talk during the lunch with one of my co-workers about some IT trending topic which I had been propagated a for a few months. During this talk I said that a platform would be awesome where users can enter their prophecies and verify prophecies of other users if they had occured and how exactly the prophecy matches with the occurence.

Due to the number of prophecies made and the number of verified prophecies you could calculate the relevancy of a prophet (= a person who makes prophecies) which indicates the expert level of the prophet. A higher number of verified prophets means that you have more expertise in a given topic than other users.

Possible customers for social forecasting networks

The first intention was to have a social network for selfish reasons. Through some mechanisms, like not being allowed to change a prophecy after someone has voted for it, you would have been pinned to one statement which could be falsified or verified. If you were right, you could always use the classical phrase: “I said so.”.

In larger companies you could identify hidden champions or motivate people to be more interested in their expert knowledge. One day I had a talk with my bank account manager who were highly interested in the project because of obvious reasons. The software would allow them to evaluate the efficency of share brokers without using real money.

Another possible target group were whistleblowers or persons who wanted to make sure that a prophecy would be published on a specific date. For this I implemented some functionality to encrypt the content of the prophecy with symmetric keys. The keys could be stored on remote servers so that only the prophet was in control of when the prophecy can be published. After Snowedens revelations I instantly thought about this feature again.

 

I have to admit that the project has one big flaw: making self-fulfilling prophecies like: “I prophecy that the share price of company VWXYZ will drop in the next few days.” If you are already an expert in your area, there is a high chance that other follower will react to this prophecy and sell their shares. The share price will drop and your prophecy could be verified… You get the idea.

Technical background

At first I started with Spring MVC but after some weeks I switched to PHP/Zend Framework 1.x/MySQL. Most of the statistical computation (relevancy of prophets, influence of prophets and so on) and the social network aspect (who follows whom, which prophecies I can see) is done through database views which made the implementation inside the services really easy.
The encryption part called remote-credential-loader (RCL) is written in Node.js. RCL polls every few minutes the deposited decryption key URLs for encrypted prophecies. To a given timestamp (e.g. five minutes before releasing the prophecy) the URL must provide the AES decryption key, otherwise the prophecy is evaluated as false.

For the frontend I used Twitter Bootstrap 2.

The whole background documentation (processes, data model, computation) I had written in LaTeX (German language only).

Current status of the project

After thinking about the idea for years I finished the beta within the scope of my Bachelor project in the year 2012. The professor who belongs to the statistical faculty and who had observed the project was really impressed about it. Since December 2012 I am the owner of prophetr.de and prophetr.com which were intended to host the social network, but it is a classical 80%/20% project status. The application misses LDAP authorization and synchronization for usage in enterprise environments, the user interface and design is pragmatical and not very user friendly and so on.

A few months after I finished the Bachelor project I read an article in the c’t. If remember correctly they were from Austria and got a lot of money for building a social forecasting network like mine. This was more or less the reason why I had abandoned the project for the last two years.

Drop me a line at me[at]schakko[dot]de if you are interested in more information.

ExceptionHandler of @ControllerAdvice is not executed

It happened again: after writing about some issues caused by different JVM class-loader order a similar problem occured on Friday. One of my collagues (Dev-A) asked me to look into a problem the team had. Because of unknown reasons the Spring Boot based application did not return a serialized JSON error object after a @Valid annotated controller method parameter had been validated.

@Controller
public class MyController {
	// Validator for MyDto (MyDtoValidator) got called
	@RequestMapping("/validate")
	public @ResponseBody MyData myMethod(@Valid MyDto myDto) {
		return new MyData()
	}
}

An @ControllerAdvice annotated class transformed any validation error into a new exception. This has been done to unify the validation errors when using Spring Data REST and Spring MVC validation.

@ControllerAdvice
public class ValidationErrorHandlerAdvice {

	private MessageSourceAccessor messageSourceAccessor;

	@Autowired
	public ValidationErrorHandlerAdvice(MessageSourceAccessor messageSourceAccessor) {
		Assert.notNull(messageSourceAccessor, "messageSourceAccessor must not be null");

		this.messageSourceAccessor = messageSourceAccessor;
	}

	@ExceptionHandler({ MethodArgumentNotValidException.class })
	@ResponseStatus(HttpStatus.BAD_REQUEST)
	@ResponseBody
	public RepositoryConstraintViolationExceptionMessage handleValidationErrors(Locale locale,
			MethodArgumentNotValidException exception) {
		// this method should be called if the validation of MyController.myMethod had failed
		return produceException(exception.getBindingResult());
	}

	@ExceptionHandler({ BindException.class })
	@ResponseStatus(HttpStatus.BAD_REQUEST)
	@ResponseBody
	public RepositoryConstraintViolationExceptionMessage handleValidationErrors(Locale locale,
			BindException exception) {
		return produceException(exception.getBindingResult());
	}

	private RepositoryConstraintViolationExceptionMessage produceException(BindingResult bindingResult) {
		return new RepositoryConstraintViolationExceptionMessage(
				new RepositoryConstraintViolationException(bindingResult), messageSourceAccessor);
	}
}

All in all, the controller advice itself looked fine to me, especially as the code is easy to understand and has been used in other projects too without any problems.

Empty HTTP response body

Nevertheless the behavior was mysterious:

  • When calling /validated in the browser, the custom validator for MyDto so the controller method got definitely hit. Nevertheless none of the exception handlers in the ValidationErrorHandlerAdvice got called. To make it more mysterious the HTTP response Spring generated did only consist of the HTTP status code 400 (Bad Request) without any character in the HTTP response body. The response body was completely clear.
  • Another developer (Dev-B) uses Linux as operating system. On his machine the code above worked without any problems and returned the expected HTTP status code 400 with the serialized JSON validation error object.

Dev-A has a Windows based machine. When he had called the “/validated” endpoint on Dev-Bs host the repsonse body contained the serialized validation error. In return, when Dev-B (Linux) had called “/validated” on Dev-As machine (Windows) the response body was empty.
I checked the HTTP request headers of both browsers but they were more or less the same and did not have any influence on any HTTP pre-filters Spring had registered. Both environments uses the Oracle JDK with different update releases (u43 vs. u63). Patching both JDKs to the same level I wanted to try at last as it seemed unlikely to be the reason.

Debugging session

I started to debug through the Spring Framework and realized that the order in which the registered exception handlers got checked for their responsibility of the current occured exception was completely different. On Dev-Bs machine the ValidationErrorHandlerAdvice were the first in the list, on Dev-A the first responsible exception handler was located in ResponseEntityExceptionHandler.
After stepping further through ResponseEntityExceptionHandler it made absolutely sense that the response body was empty on Dev-As machine. But it does not made any sense that the ResponseEntityExceptionHandler got loaded in the first place.

After searching for more @ControllerAdvice annotated classes in the project I found this piece of code:

@ControllerAdvice
public class CustomErrorController extends ResponseEntityExceptionHandler {
	@ExceptionHandler()
	public ModelAndView notFound(HttpServletRequest req, Exception exception) {
		LOG.info(exception.getMessage());
		ModelAndView mav = new ModelAndView();
		// ... not so important ...
		return mav;
	}
}

Okay, at least the exception handler of ResponseEntityExceptionHandler was introduced without any Spring magic.

Fixing the problem

During debugging the initialization phase of Spring I saw that the order of the detected controller advices was different between both systems: CustomErrorController got registered before ValidationErrorHandlerAdvice on Dev-A and vice versa on Dev-B. As the wrong behavior only occured on Windows machines I assume that the underlying component scan is responsible for the different order.

In the end the fix for this solution was easy. I annotated both controllers with @Order and gave the ValidationErrorHandlerAdvice a higher precedence than CustomErrorController.

How to fix NoSuchMethodError or NoSuchMethodException

Yesterday my team had the situation that a deployment failed with a NoSuchMethodError, specifically the method com/google/common/collect/ImmutableList.copyOf could not be found while querying the Confluence REST API.

NoSuchMethodEror and NoSuchMethodException occur of obvious reasons: a method should be called during runtime but the providing class does not contain the method.

NoSuchMethodExceptions is thrown when the JVM tries to make a call to a method through Java Reflection API. NoSuchMethodError is thrown when the compiled Java code directly calls the method without using the Reflection API.
Because of its nature the reason for a NoSuchMethodException can be a syntactical issue (e.g.misspelled method name in getDeclaredMethod). If you receive the exception during development, please check the correct spelling of the method name you try to call through reflection.

There are mostly two reasons why this error occurs during runtime:

  • The method signature (method name and expected parameters) does exist nowhere in your classpath. There could be an issue in your deployment / packaging phase. For a simple web project which is packaged through Maven this is very unlikely. But if you try to use overlays with classes outside of your POM definition, there could your problem be located.
  • The method signature does exist mulitple times in your classpath. It means, you have different versions of the class in your classpath. The classes could have the same method names but can differ in the parameter list.
    It highly depends upon on the environment which of the classes in JAR files have precedence. There is no such JVM specification that a classloader has to either fetch JARs in alphabetical or last-touched order or use a first-come/first-serve last-come/first-serve order. For example, JAR files are loaded in Tomcat until <= 7 in alphabetical order. Tomcat 8 let the filesystem make the decision which JAR comes first (Order of loading jar files from lib directory).

To identify the source of the problem, navigate to the main classpath directory of your application (e.g. WEB-INF/lib) and execute

for jar in *.jar; do for class in $(jar -tf $jar | grep $CLAZZ.class | sed 's/.class//g'); do javap -classpath $jar -s $class | grep -A 2 $METHOD && echo $jar.$class; done; done

Replace $CLAZZ with the name of the class and $METHOD with the name of the method. The shell script above searches for every occurence of the method inside any of the JARs and prints out the different signatures.

  • If there is no result, you hit the first case: your deployment script did not include the required dependency.
  • If there are multiple results from different JAR files, you have to compare the stacktrace of your application logs with the output of the script. Check the dependency hierarchy of your Maven POM and exclude the version not containing the expected method signature.

In our case, I had mistakenly included google-collections-1.0 and guava-11.0.2 in a referenced JAR which both provide ImmutableList. google-collection is the older dependency and does not contain the copyOf method. In the development environment, the (Spring Boot) application has been always executed through the embedded application server. In production, the WAR was deployed inside a Tomcat 8 container. In the end we removed the google-collections from the referenced JAR and the issue has been fixed.

One last word from the Tomcat Bugzilla by Mark Thomas:

Applications that depend on JARs being searched for classes in a particular order are broken and should be fixed.

Collecting and visualizing metrics with statsd, InfluxDB and Grafana on Fedora 22

My employer NeosIT offers a web based SMS notifiyng solution for organizations with security roles named ZABOS. In the last months we extended the ZABOS application to support digital alerting through POCSAG. After some problems with a third party component we implemented the ability to collect all POCSAG telegrams delivered in the near circumcircle and to notify the authorized recipients by SMS. Most of the incoming telegrams are discarded because they are not assigned in our database. But nevertheless I was interested in graphical representation of all incoming POCSAG messages, and additionally in a comparision to alerts sent with ZVEI, an analogue notification protocol. The ZABOS application log file contains all relevant information, which I wanted to extract.

Setting the stage

Our current infrastructure is based upon Fedora systems and some CentOS boxes. A central Logstash server collects incoming log messages through the Lumberjack input filter. After reviewing possible alternatives I had decided to implement statsd, InfluxDB and Grafana.

InfluxDB

InfluxDB is an open-source distributed time-series database which stores points in time and assigns key/values to it. Installing it on Fedora 22 is easy: get the latest RPM, install it and open the TCP ports:

wget https://s3.amazonaws.com/influxdb/influxdb-0.9.4.2-1.x86_64.rpm
sudo dnf install ./influxdb-0.9.4.2-1.x86_64.rpm</code>

# open network ports
# 8083: admin GUI port
sudo firewall-cmd --add-port=8083/tcp --permanent
# 8086: REST API
sudo firewall-cmd --add-port=8086/tcp --permanent
sudo firewall-cmd --reload

systemctl start influxdb
journalctl -f -u influxdb

After installing the RPM, navigate to http://localhost:8083 and set up a new database. The screenshots in the official documentation are slightly outdated, so use the query input:

CREATE DATABASE "demo"
CREATE USER "demo" WITH PASSWORD 'demo'

Make sure, that you can although open the URL http://localhost:8086/ping. It should return a valid HTTP 204 response.

statsd

statsd is node.js service which collects time series, provided through UDP or TCP. Most producers, e.g. Logstash, provide a statsd interface. statsd itself is pluggable and has a backend plug-in for InfluxDB. Every incoming time series is forwarded to the InfluxDB instance.

# get required packages
sudo dnf install nodejs npm git
cd /opt
sudo git clone https://github.com/etsy/statsd.git
cd statsd

# download InfluxDB backend
npm install statsd-influxdb-backend -d

# open network ports
firewall-cmd --add-port=8125/tcp --permanent
firewall-cmd --add-port=8125/udp --permanent
firewall-cmd --reload

# make configuration directory an copy example configuration
mkdir /etc/statsd/
cp exampleConfig.js /etc/statsd/config.js

# create a user
adduser statsd
# add systemd unit
vi /etc/systemd/system/statsd.service

The statsd.service file contains the unit definition for systemd. I mostly used the sample given at digitalocean.com:

[Service]
ExecStart=/usr/bin/node /opt/statsd/stats.js /etc/statsd/config.js
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=statsd
User=statsd
Group=statsd
Environment=NODE_ENV=production</code>

[Install]
WantedBy=multi-user.target

After saving the unit definition, edit the /etc/statsd/config.js:

{
influxdb: {
version: 0.9, // !!! we installed 0.9
host: '127.0.0.1', // InfluxDB host. (default 127.0.0.1)
port: 8086, // InfluxDB port. (default 8086)
database: 'demo', // InfluxDB database instance. (required)
username: 'demo', // InfluxDB database username. (required)
password: 'demo', // InfluxDB database password. (required)
flush: {
enable: true // Enable regular flush strategy. (default true)
},
proxy: {
enable: false, // Enable the proxy strategy. (default false)
suffix: 'raw', // Metric name suffix. (default 'raw')
flushInterval: 1000 // Flush interval for the internal buffer.
// (default 1000)
}
},
port: 8125, // StatsD port.
backends: ['./backends/console', 'statsd-influxdb-backend'],
debug: true,
legacyNamespace: false
}

If you miss the version property, statsd-influxdb-backend uses the old protocol version. 0.9 is incompatible with prior versions, so you will receive HTTP 404 errors after forwarding messages from statsd to InfluxDB.

# enable service
systemctl enable statsd
systemctl start statsd

journcalctl -f -u statsd

Logstash

In our special case I had to use the logstash-forwarder to forward the ZABOS application log to Lumberjack. To be compatible with our existing Logstash infrastructure, I configured a special input filter to extract POCSAG RICs and ZVEI series from the ZABOS log file. The filter itself is out of scope of this blog entry.

The statsd output filter for Logstash provides the ability to send extracted log data to statsd. The configuration is straight forward:

filter {
 file {
  # log extraction logic skipped
 }
}

output {
 if [pocsag_subric] {
  statsd {
   host => "127.0.0.1"
   port => 8125
   increment => "pocsag.incoming.%{pocsag_subric}"
  }
 }
}

This conditional output filter increments a key with the given POCSAG SubRIC if a pocsag_subric field is not empty.

After manual running the Logstash agent with the configuration above, Logstash sends all found POCSAG SubRICs to the local statsd instance which in turn forwards it to InfluxDB.

One note about logstash-output-influxdb: it supports a direct output into InfluxDB without using statsd, but it supports only the old API piror 0.9. In addition, most time series producers are sending in a statsd format. So the setup I described is more complex but you gain an advantage in flexibility.

Grafana

At this point I was able to forward all POCSAG telegrams to InfluxDB. To visualize the collected information, I installed Grafana. The Grafana client connects to different backend databases like OpenTSDB, ElasticSearch and InfluxDB to produce time series based graphs. Installing Grafana can be accomplished with yum/dnf:

sudo dnf install https://grafanarel.s3.amazonaws.com/builds/grafana-2.1.3-1.x86_64.rpm</code>

# open ports
firewall-cmd --add-port=3000/tcp --permanent
firewall-cmd --reload

systemctl enable grafana-server
systemctl start grafana-server

After navigating to http://localhost:3000 you need to set up a new data source: Click on Data Sources > Add New and enter the credentials to your InfluxDB instance.

Important information: You have to enter a FQDN as database URL and not http://localhost! Your browser will directly connect to the InfluxDB backend, so your browser must have access to the InfluxDB REST endpoint.

If you need professional consulting or development services for the topics above, just look on our website or leave us a message at info[at]neos-it[dot]de.