FAT JAR files for testing in restricted test environments
Sometimes we have very restricted test environment where there is no possibility to get direct access to the code repository. FAT JAR files can be useful solution of such problems. Here I present 3 possible solutions.
Formulation of the problem
There is a testing environment where there is no direct access to the code repository. We can however (after security scan) send some files to this environment.
We have some tests written in Java, and we want to execute them in the restricted environment where SUT is installed.
Our tests without restrictions could be run with Maven, but it is impossible in the environment because either Maven is not installed there nor it has no access to the repositories with libraries.
Fortunately in the environment we have Java on board, so we want to collect everything what is needed to execute tests (with all dependencies) and send it all together to the environment and just run it as a Java process (without any Maven).
Where such situations appear? One possibility is that on the environment are stored very confidential data, and the access is extremely limited. Another possibility is that the environment is stored in some cloud with no connectivity to the repository.
Solutions
Solution 1 — Maven Assembly Plugin
The solution consists of the following elements:
- entries in
pom.xml assembly.xmlfileMainClassclass withmainmethod- changes in the code necessary to make java read configuration files
First we have to add the plugin
<plugin>
<artifactid>maven-assembly-plugin</artifactid>
<version>2.5.4</version>
<configuration>
<descriptor>src/tests/resources/assembly.xml</descriptor>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<archive>
<manifest>
<mainclass>com.example.tests.MainClass</mainclass>
</manifest>
</archive>
</configuration>
</execution>
</executions>
</plugin>Then we have to add entry which all XML files with test suites and all test resources will place in the build directory. We can do it in the following way:
<testresource>
<filtering>true</filtering>
<directory>${project.basedir}/suites</directory>
<targetpath>${project.build.directory}/suites</targetpath>
<includes>
<include>*.xml</include>
</includes>
</testresource>
<testresource>
<filtering>true</filtering>
<directory>${project.basedir}/src/tests/resources</directory>
<targetpath>${project.build.directory}/test-resources</targetpath>
<includes>
<include>*.*</include>
</includes>
</testresource>We create assembly.xml file
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<id>fat-tests</id>
<formats>
<format>jar</format>
</formats>
<includebasedirectory>false</includebasedirectory>
<dependencysets>
<dependencyset>
<outputdirectory>/</outputdirectory>
<useprojectartifact>true</useprojectartifact>
<unpack>true</unpack>
<scope>test</scope>
</dependencyset>
</dependencysets>
<filesets>
<fileset>
<directory>${project.build.directory}/test-classes</directory>
<outputdirectory>/</outputdirectory>
<includes>
<include>**/*.class</include>
<include>**/*.json</include>
<include>**/*.xml</include>
</includes>
<usedefaultexcludes>true</usedefaultexcludes>
</fileset>
<fileset>
<directory>${project.build.directory}/suites</directory>
<outputdirectory>./</outputdirectory>
<includes>
<include>**/*.xml</include>
</includes>
<usedefaultexcludes>true</usedefaultexcludes>
</fileset>
</filesets>
</assembly>At last we add MainClass class with main method. Let us assume that we have also the file testng.xml in the resources. We take it from there and execute. This may be also easily extended for cases with multiple suites and external arguments support.
public class Main {
public static void main(String[] args) throws URISyntaxException {
TestNG testng = new TestNG();
URI resource = Main.class.getClassLoader().getResource("testng.xml")
.toURI();
setTestSuites(testng, resource);
testng.setTestSuites(Lists.newArrayList());
testng.addListener(new TestListenerAdapter());
testng.run();
}
/**
* https://www.programcreek.com/java-api-examples/?code=opengeospatial/teamengine/teamengine-master/teamengine-spi/src/main/java/com/occamlab/te/spi/executors/testng/TestNGExecutor.java
* <p>
* Sets the test suite to run using the given URI reference. Three types of
* references are supported:
* </p><ul>
* <li>A file system reference</li>
* <li>A file: URI</li>
* <li>A jar: URI</li>
* </ul>
*
* @param driver The main TestNG driver.
* @param ets A URI referring to a suite definition.
*/
private static void setTestSuites(TestNG driver, URI ets) {
if (ets.getScheme().equalsIgnoreCase("jar")) {
// jar:{url}!/{entry}
String[] jarPath = ets.getSchemeSpecificPart().split("!");
File jarFile = new File(URI.create(jarPath[0]));
driver.setTestJar(jarFile.getAbsolutePath());
driver.setXmlPathInJar(jarPath[1].substring(1));
} else {
List<string> testSuites = new ArrayList<string>();
File tngFile = new File(ets);
if (tngFile.exists()) {
System.out.printf("Using TestNG config file %s", tngFile.getAbsolutePath());
testSuites.add(tngFile.getAbsolutePath());
} else {
throw new IllegalArgumentException("A valid TestNG config file reference is required.");
}
driver.setTestSuites(testSuites);
}
}
}One of the drawbacks of this solution is that whenever we are accessing in the code to the resources we have to implement similar hacks or at least revise the code if it can handle accessing files from jar. This not always have to be so simple.
For example sometimes you have to dig into libraries and do something like this:
public static <t> T read(URL resource, Class<t> tClass) throws IOException {
LOGGER.debug("Reading resource: " + resource);
try {
if (resource.toURI().getScheme().equalsIgnoreCase("jar")) {
// jar: {url}!/{entry}
String[] jarPath = resource.toURI().getSchemeSpecificPart().split("!");
return OBJECT_MAPPER.readValue(
ClassLoader.getSystemClassLoader().getResourceAsStream(jarPath[1].substring(1),
tClass);
} else {
return OBJECT_MAPPER.readValue(resource, tClass);
}
} catch (IOException | URISyntaxException e) {
throw new IOException("Unable to read file: '" + resource + "'.", e);
}
}Classes with the same name and package might exist among your dependencies. Which one will end up in our fat jar? This sometimes may be a problem.
Solution 2 — Maven Shade Plugin
Apache Maven Shade Plugin also provides the capability to package the artifacts in an FAT JAR file, which consists of all dependencies required to run the project. Moreover it comes with the concept of “transformers”, which gives ability to merge conflicting files together in the monolithic JAR rather than having one overwrite the other. Especially important is possibility to renaming packages of some dependencies.
This may be useful for Apache CXF framework for JAX-WS. (See here for more details)
<plugins>
<groupid>org.apache.maven.plugins</groupid>
<artifactid>maven-jar-plugin</artifactid>
<version>3.1.0</version>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
<configuration>
<classesdirectory>src/tests/java</classesdirectory>
<archive>
<manifest>
<addclasspath>true</addclasspath>
<mainclass>com.example.tests.MainClass</mainclass>
</manifest>
</archive>
</configuration>
</plugins>
<plugins>
<groupid>org.apache.maven.plugins</groupid>
<artifactid>maven-shade-plugin</artifactid>
<version>1.4</version>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createdependencyreducedpom>true</createdependencyreducedpom>
<shadedartifactattached>true</shadedartifactattached>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<artifactset>
<includes>
<include>samples:*</include>
</includes>
</artifactset>
</configuration>
</execution>
</executions>
</plugins>The main difference is that Maven Shade Plugin uses the JAR generated by the Maven Jar Plugin and adds dependencies to it. This strategy has to be taken into account not only by adding plugins to the pom.xml but also in execution. To generate FAT JAR file you have to execute both packageand shade:shade i.e.
mvn clean package shade:shadeThis solution sounds cool. Right? Where is the trick? By using this solution we are trying to have the classes from both src/main/java and src/test/java in one directory and then in the jar. What will happen if you have two different classes?
Additionally, both Solution 1 and Solution 2 flatten the structure of dependencies. In complex project it may cause some problems with clashing classes. Keep also in mind that accessing resources here looks exactly the same as in the Solution 1, so it causes the same problems.
Solution 3 — Maven Assembly Plugin once again but differently
This solution is using the same plugin as the Solution 1 but in this case we do not unpack jars from Maven repository. We take them as they are and pack them as separate jars into the bigger jar.
In the environment we have to unpack everything, so all dependencies will present exactly as they were taken from the repository.
In this way we can avoid problems with clashing dependencies and with reflection calls.
How it works in details? The solution consists of the following elements:
- moving your test from main project to a module (if it is not yet done) named e.g.
tests - adding a module in the project named e.g.
tests-assembly - entry in
pom.xmlin the moduletests - entries in
pom.xmlin the moduletests-assembly assembly.xmlfile in\src\main\assembly(javadirectory is not necessary insrc) oftests-assemblymodule
For purpose of our example let us assume that we have the following modules in the main Maven project.
<modules>
<module>tests-assembly</module>
<module>tests</module>
</modules>In the tests module in the pom.xml in the 'build' part we add the following entry. It is responsible for packing our tests.
<testsourcedirectory>${project.basedir}/src/test/java</testsourcedirectory>
<testoutputdirectory>${project.build.directory}/tests-classes</testoutputdirectory>
<plugins>
<plugin>
<groupid>org.apache.maven.plugins</groupid>
<artifactid>maven-jar-plugin</artifactid>
<version>3.2.2</version>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>In the module tests-assembly the file pom.xml can look as follows.
<!--?xml version="1.0" encoding="UTF-8"?-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelversion>4.0.0</modelversion>
<parent>
<groupid>com.test.classes</groupid>
<artifactid>testng-maven</artifactid>
<version>1.0-SNAPSHOT</version>
</parent>
<artifactid>tests-assembly</artifactid>
<properties>
<maven.compiler.source>18</maven.compiler.source>
<maven.compiler.target>18</maven.compiler.target>
<project.build.sourceencoding>UTF-8</project.build.sourceencoding>
</properties>
<dependencies>
<dependency>
<groupid>com.test.classes</groupid>
<artifactid>tests</artifactid>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<testresources>
<testresource>
<filtering>true</filtering>
<directory>../tests/src/test/resources</directory>
<targetpath>${project.build.directory}/resources</targetpath>
</testresource>
</testresources>
<plugins>
<plugin>
<artifactid>maven-assembly-plugin</artifactid>
<version>3.1.1</version>
<configuration>
<descriptors>src/main/assembly/assembly.xml</descriptors>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<archive>
<manifest>
<mainclass>org.testng.TestNG</mainclass>
</manifest>
</archive>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>It is necessary to add the main project test-jar as dependency. We want pack this jar with tests from the main project into the bigger jar.
The build section specifies test resources location, plugin we wan to use, assembly.xml file location and the main class.
The last thing is the assembly.xml file.
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<id>fat-tests</id>
<formats>
<format>jar</format>
</formats>
<includebasedirectory>false</includebasedirectory>
<dependencysets>
<dependencyset>
<outputdirectory>lib/</outputdirectory>
<useprojectartifact>true</useprojectartifact>
<useprojectattachments>true</useprojectattachments>
<scope>test</scope>
</dependencyset>
</dependencysets>
<filesets>
<fileset>
<directory>${project.build.directory}/resources</directory>
<outputdirectory>resources/</outputdirectory>
<includes>
<include>**/*.xml</include>
</includes>
<usedefaultexcludes>false</usedefaultexcludes>
</fileset>
</filesets>
</assembly>In this file we just tell, that we want to have all dependency jars in the lib directory and all test resources in resources directory.
Ok, how to use it? First we have to prepare jar with our main project tests.
mvn clean installWe have to do it because it generates the jar which will be packed in the next step. Such a jar is placed in the local Maven repo.
Then, we are executing the following for the module ‘tests-assembly’.
mvn clean packageThis step is crucial. All dependencies (including the result of previous step) are packed into a bigger jar.
You will get something like tests-assembly-1.0-SNAPSHOT-fat-tests.jar. Such a jar is not ready for execution, but it is ready to be sent to the destination.
Let us assume that after thorough security scans we successfully transferred this file to the test environment. Now it is a time for making use of the file. It has to be unpacked in the following way.
jar xf ./tests-assembly/target/tests-assembly-1.0-SNAPSHOT-fat-tests.jarFinally, you can just run everything
java -cp 'lib/*' org.testng.TestNG ./resources/testng.xmlEverything above can be easily scripted and automated.
Of course, you can pass any arguments, but keep in mind that there is no Maven here, so you cannot use Maven profiles. If you have them in your project, look into their definition and pass appropriate Java properties.
Other possibilities not based on FAT JAR
If FAT JAR is not an option then there is another possibility — containers. One can pack tests into container and then transfer whole container into the test environment. However, it is rather not very likely that it would be easier than transferring FAT JAR files.
Summary
In my opinion, Solution 3 is the most robust and elegant option among the available solutions. Additionally, it is quite simple to remove the module when it is no longer needed, and there are no modifications required to the existing code.
It is true that jars are not typically intended to be used in this manner, and the idea of treating them as simple archives may not seem particularly refined. However, there is nothing inherently wrong with this approach. Using zip files would achieve a similar outcome, but the process of doing so would be more cumbersome and require additional effort.
The story was originally created by me, but it may contain parts that were created with AI assistance. My original text has been corrected and partially rephrased by Chat Generative Pre-trained Transformer to improve the language.
