diff --git a/README.ftl.md b/README.ftl.md new file mode 100644 index 0000000000..c7ca7433d9 --- /dev/null +++ b/README.ftl.md @@ -0,0 +1,119 @@ +<#assign project_id="gs-batch-processing"> + +# Getting Started: Creating a Batch Service + +What you'll build +----------------- + +This guide walks you through creating a basic batch-driven solution. You build a service that imports data from a CSV spreadsheet, transforms it with custom code, and stores the final results in a database. + +What you'll need +---------------- + + - About 15 minutes + - <@prereq_editor_jdk_buildtools/> + +## <@how_to_complete_this_guide jump_ahead='Create a business class'/> + + + +Set up the project +------------------ +<@build_system_intro/> + +<@create_directory_structure_hello/> + +### Create a Maven POM + + <@snippet path="pom.xml" prefix="complete"/> + +<@bootstrap_starter_pom_disclaimer/> + +### Create business data + +Typically your customer or a business analyst supplies a spreadsheet. In this case, you make it up. + + <@snippet path="src/main/resources/sample-data.csv" prefix="initial"/> + +This spreadsheet contains a first name and a last name on each row, separated by a comma. This is a fairly common pattern that Spring handles out-of-the-box, as you will see. + +### Define the destination for your data + +Next, you write a SQL script to create a table to store the data. + + <@snippet path="src/main/resources/schema.sql" prefix="initial"/> + + +Create a business class +----------------------- + +Now that you see the format of data inputs and outputs, you write code to represent a row of data. + + <@snippet path="src/main/java/hello/Person.java" prefix="complete"/> + +You can instantiate the `Person` class either with first and last name through a constructor, or by setting the properties. + +Create an intermediate processor +-------------------------------- + +A common paradigm in batch processing is to ingest data, transform it, and then pipe it out somewhere else. Here you write a simple transformer that converts the names to uppercase. + + <@snippet path="src/main/java/hello/PersonItemProcessor.java" prefix="complete"/> + +`PersonItemProcessor` implements Spring Batch's `ItemProcessor` interface. This makes it easy to wire the code into a batch job that you define further down in this guide. According to the interface, you receive an incoming `Person` object, after which you transform it to an upper-cased `Person`. + +> **Note:** There is no requirement that the input and output types be the same. In fact, after one source of data is read, sometimes the application's data flow needs a different data type. + +Put together a batch job +---------------------------- + +Now you put together the actual batch job. Spring Batch provides many utility classes that reduce the need to write custom code. Instead, you can focus on the business logic. + + <@snippet path="src/main/java/hello/BatchConfiguration.java" prefix="complete"/> + +For starters, the `@EnableBatchProcessing` annotation adds many critical beans that support jobs and saves you a lot of leg work. + +Break it down: + + <@snippet "src/main/java/hello/BatchConfiguration.java" "readerwriterprocessor" "/complete"/> + +The first chunk of code defines the input, processor, and output. +- `reader()` creates an `ItemReader`. It looks for a file called `sample-data.csv` and parses each line item with enough information to turn it into a `Person`. +- `processor()` creates an instance of our `PersonItemProcessor` you defined earlier, meant to uppercase the data. +- `write(DataSource)` creates an `ItemWriter`. This one is aimed at a JDBC destination and automatically gets a copy of the dataSource created by `@EnableBatchProcessing`. It includes the SQL statement needed to insert a single `Person` driven by java bean properties. + +The next chunk focuses on the actual job configuration. + + <@snippet "src/main/java/hello/BatchConfiguration.java" "jobstep" "/complete"/> + +The first method defines the job and the second one defines a single step. Jobs are built from steps, where each step can involve a reader, a processor, and a writer. + +In this job definition, you need an incrementer because jobs use a database to maintain execution state. You then list each step, of which this job has only one step. The job ends, and the java API produces a perfectly configured job. + +In the step definition, you define how much data to write at a time. In this case, it writes up to ten records at a time. Next, you configure the reader, processor, and writer using the injected bits from earlier. + +> **Note:** chunk() is prefixed `` because it's a generic method. This represents the input and output types of each "chunk" of processing, and lines up with `ItemReader` and `ItemWriter`. + +Finally, you run the application. + + <@snippet "src/main/java/hello/BatchConfiguration.java" "templatemain" "/complete"/> + +This example uses a memory-based database (provided by `@EnableBatchProcessing`), meaning that when it's done, the data is gone. For demonstration purposes, there is extra code to create a `JdbcTemplate`, query the database, and print out the names of people the batch job inserts. + +## <@build_an_executable_jar/> + + +Run the batch job +----------------- + +Now you can run the job from the JAR as well, and distribute that as an executable artifact: + + $ java -jar target/${project_id}-complete-0.1.0.jar + + +The job prints out a line for each person that gets transformed. After the job runs, you can also see the output from querying the database. + +Summary +------- + +Congratulations! You built a batch job that ingested data from a spreadsheet, processed it, and wrote it to a database. diff --git a/README.md b/README.md index 863f1e659d..05f55db9ee 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,4 @@ + # Getting Started: Creating a Batch Service What you'll build @@ -26,11 +27,13 @@ To **start from scratch**, move on to [Set up the project](#scratch). To **skip the basics**, do the following: - [Download][zip] and unzip the source repository for this guide, or clone it using [git](/understanding/git): -`git clone https://github.com/springframework-meta/{@project-name}.git` - - cd into `{@project-name}/initial` - - Jump ahead to [Create a resource representation class](#initial). +`git clone https://github.com/springframework-meta/gs-batch-processing.git` + - cd into `gs-batch-processing/initial` + - Jump ahead to [Create a business class](#initial). + +**When you're finished**, you can check your results against the code in `gs-batch-processing/complete`. +[zip]: https://github.com/springframework-meta/gs-batch-processing/archive/master.zip -**When you're finished**, you can check your results against the code in `{@project-name}/complete`. Set up the project @@ -56,7 +59,7 @@ In a project directory of your choosing, create the following subdirectory struc 4.0.0 org.springframework - gs-batch-processing-initial + gs-batch-processing-complete 0.1.0 @@ -76,6 +79,19 @@ In a project directory of your choosing, create the following subdirectory struc + + hello.BatchConfiguration + + + + + + org.apache.maven.plugins + maven-shade-plugin + + + + spring-snapshots @@ -100,15 +116,14 @@ In a project directory of your choosing, create the following subdirectory struc TODO: mention that we're using Spring Bootstrap's [_starter POMs_](../gs-bootstrap-starter) here. -> Note to experienced Maven users who don't use an external parent project: You can take out the project later, it's just there to reduce the amount of code you have to write to get started. - +Note to experienced Maven users who are unaccustomed to using an external parent project: you can take it out later, it's just there to reduce the amount of code you have to write to get started. ### Create business data Typically your customer or a business analyst supplies a spreadsheet. In this case, you make it up. `src/main/resources/sample-data.csv` -```text +```csv Jill,Doe Joe,Doe Justin,Doe @@ -423,30 +438,37 @@ Finally, you run the application. This example uses a memory-based database (provided by `@EnableBatchProcessing`), meaning that when it's done, the data is gone. For demonstration purposes, there is extra code to create a `JdbcTemplate`, query the database, and print out the names of people the batch job inserts. +### Build an executable JAR -Build an executable JAR ------------------------ -Add the following to your `pom.xml` file, keeping existing properties and plugins intact: +Now that your `Application` class is ready, you simply instruct the build system to create a single, executable jar containing everything. This makes it easy to ship, version, and deploy the service as an application throughout the development lifecycle, across different environments, and so forth. + +Add the following configuration to your existing Maven POM: `pom.xml` ```xml - - hello.BatchConfiguration - + + hello.Application + - - - - org.apache.maven.plugins - maven-shade-plugin - - - + + + + org.apache.maven.plugins + maven-shade-plugin + + + ``` -Create a single executable JAR file containing all necessary dependency classes: +The `start-class` property tells Maven to create a `META-INF/MANIFEST.MF` file with a `Main-Class: hello.Application` entry. This entry enables you to run the jar with `java -jar`. - $ mvn package +The [Maven Shade plugin][maven-shade-plugin] extracts classes from all jars on the classpath and builds a single "über-jar", which makes it more convenient to execute and transport your service. + +Now run the following to produce a single executable JAR file containing all necessary dependency classes and resources: + + mvn package + +[maven-shade-plugin]: https://maven.apache.org/plugins/maven-shade-plugin Run the batch job @@ -463,5 +485,3 @@ Summary ------- Congratulations! You built a batch job that ingested data from a spreadsheet, processed it, and wrote it to a database. - -[zip]: https://github.com/springframework-meta/gs-batch-processing/archive/master.zip diff --git a/complete/pom.xml b/complete/pom.xml index 3348c3f432..c5b5d7dcac 100644 --- a/complete/pom.xml +++ b/complete/pom.xml @@ -24,7 +24,6 @@ - hello.BatchConfiguration @@ -37,7 +36,6 @@ - diff --git a/complete/src/main/java/hello/BatchConfiguration.java b/complete/src/main/java/hello/BatchConfiguration.java index 1dd67ae470..5c995024c4 100644 --- a/complete/src/main/java/hello/BatchConfiguration.java +++ b/complete/src/main/java/hello/BatchConfiguration.java @@ -35,7 +35,7 @@ import org.springframework.jdbc.core.RowMapper; @EnableAutoConfiguration public class BatchConfiguration { - // {!begin#reader-writer-processor} + // {!begin readerwriterprocessor} @Bean public ItemReader reader() { FlatFileItemReader reader = new FlatFileItemReader(); @@ -64,9 +64,9 @@ public class BatchConfiguration { writer.setDataSource(dataSource); return writer; } - // {!end#reader-writer-processor} + // {!end readerwriterprocessor} - // {!begin#job-step} + // {!begin jobstep} @Bean public Job importUserJob(JobBuilderFactory jobs, Step s1) { return jobs.get("importUserJob") @@ -86,9 +86,9 @@ public class BatchConfiguration { .writer(writer) .build(); } - // {!end#job-step} + // {!end jobstep} - // {!begin#template-main} + // {!begin templatemain} @Bean public JdbcTemplate jdbcTemplate(DataSource dataSource) { return new JdbcTemplate(dataSource); @@ -106,5 +106,5 @@ public class BatchConfiguration { System.out.println("Found <" + person + "> in the database."); } } - // {!end#template-main} + // {!end templatemain} }