Apache Spark is an open source, general-purpose distributed computing engine used for processing and analyzing a large amount of data. Hence, adding sequential and unique IDs to a Spark Dataframe is not very straight forward, because of distributed nature of it.
In a relational database, locking refers to actions taken to prevent data from changing between the time it is read and the time is used.
The curl is a command line tool used to download or upload data to a server via supported protocols such as HTTP, FTP, IMAP, SFTP, TFTP, IMAP, POP3, SCP, etc. It is a remote utility, so it works without user interaction.
Annotations, a form of metadata, provide data about a program that is not part of the program itself. Annotations have no direct effect on the operation of the code they annotate. Annotations have been a powerful part of Java, but most times we tend to use them rather than create them.
An ExecutionContext is a set of key-value pairs containing information that is scoped to either StepExecution or JobExecution. Spring Batch persists the ExecutionContext, which helps in cases where you want to restart a batch run (e.g., when a fatal error has occurred, etc.). All that is needed is to put any object to be shared between steps into the context and the framework will take care of the rest. After restart, the values from the prior ExecutionContext are restored from the database and applied.
Mockito is a mocking framework. It is a Java-based library used to create simple and basic test APIs for performing unit testing of Java applications. It internally uses the Java Reflection API to generate mock objects for a specific interface. Mockito is in top 10 Java library across all libraries, not only the testing tools.
JUnit is a unit testing framework for the Java programming language. JUnit is used primarily to test each and every unit or component of your application, like classes and methods. It helps to write and run repeatable automated tests to ensure your code runs as intended.
All JavaBeans are POJOs but not all POJOs are JavaBeans.
Just add the below line in application.properties of your spring boot application and done. :)
logging.file = /path/logFileName.log
One main advantage of the Apache Spark is, it splits data into multiple partitions and executes operations on all partitions of data in parallel which allows us to complete the job faster.While working with partition data we often need to increase or decrease the partitions based on data distribution. Methods repartition and coalesce helps us to repartition.
