Augmentation patterns to modernize a mainframe on Amazon

{"value":"Customers with mainframes want to use Amazon Web Services (AWS) to increase agility, maximize the value of their investments, and innovate faster. On June 8, 2022, AWS announced the general availability of AWS Mainframe Modernization, a new service that makes it faster and simpler for customers to modernize mainframe-based workloads.\n\nIn this post, we discuss the common use cases and the augmentation architecture patterns that help liberate data from mainframe for modern data analytics, get rid of expensive and unsupported tape storage solutions for mainframe, build new capabilities that integrate with core mainframe workloads, and enable agile development and testing by adopting CI/CD for mainframe.\n\n\n### **Pattern 1: Augment mainframe data retention with backup and archival on AWS**\n\n\nMainframes process and generate the most business-critical data. It’s imperative to provide data protection via solutions, such as data backup, archiving, and disaster recovery. Mainframes usually use automated tape libraries—virtual tape libraries for backup and archive. These tapes need to be stored, organized, and transported to vaults and disaster recovery sites. All this can be very expensive and rigid.\n\nThere is a more cost-effective approach that helps simplify the operations of tape libraries: leverage AWS partner tools, such as [Model9](https://aws.amazon.com/cn/blogs/apn/how-cloud-backup-for-mainframes-cuts-costs-with-model9-and-aws/), to transparently migrate the data on tape storage to AWS.\n\nAs depicted in Figure 1, mainframe data can be transferred via the secured network connection using [AWS Transfer Family](https://aws.amazon.com/cn/aws-transfer-family/) services or [AWS DataSync](https://aws.amazon.com/cn/datasync/) to AWS cloud storage services, such as [Amazon Elastic File System](https://aws.amazon.com/cn/efs/), [Amazon Elastic Block Store](https://aws.amazon.com/cn/ebs/), and [Amazon Simple Storage Service (S3)](https://aws.amazon.com/s3/). After data is stored in AWS cloud, you can configure and move data among these services to meet with the business data processing need. Depending on data storage requirements, data storage costs can be further optimized by configuring [S3 Lifecyle](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) policies to move data among [Amazon S3 storage classes](https://aws.amazon.com/cn/s3/storage-classes/). For long-term data archiving purpose, you can choose [S3 Glacier storage class](https://aws.amazon.com/cn/s3/storage-classes/glacier/) to achieve durability, resilience, and the optimal cost effectiveness.\n\n\n\nFigure 1. Mainframe data backup and archival augmentation\n\n\n### **Pattern 2: Augment mainframe with agile development and test environments including CI/CD pipeline on AWS**\n\n\nFor any business-critical business application, a typical mainframe workload requires development and test environments to support production workloads. It’s common to see the lengthy application development lifecycle, a lack of automated testing, and an absent CI/CD pipeline with most of mainframes. Furthermore, the existing mainframe development processes and tools are outdated, as they are unable to keep up with the business pace, resulting in a growing backlog. Organizations with mainframes look for application development solutions to solve these challenges.\n\nAs demonstrated in Figure 2, AWS developer tools orchestrate code compilation, testing, and deployment among mainframe test environments. Mainframe test environments are either provided by the mainframe vendors as emulators or by AWS partners, such as [Micro Focus](https://www.microfocus.com/en-us/home). You can load the preferred developer tools and run an integrated development environment (IDE) from [Amazon WorkSpaces](Amazon WorkSpaces) or [Amazon AppStream 2.0](https://aws.amazon.com/appstream2/). Developers create or modify code in the IDE, and then commit and push their code to [AWS CodeCommit](https://aws.amazon.com/cn/codecommit/). As soon as the code is pushed, an event is generated and triggers the pipeline in [AWS CodePipeline](https://aws.amazon.com/cn/codepipeline/) to build the new code in a compilation environment via [AWS CodeBuild](https://aws.amazon.com/cn/codebuild/). The pipeline pushes the new code to the test environment.\n\nTo optimize cost, you can scale the test environment capacity to meet needs. The tests are executed, and the test environment can be shut down when not in use. When the tests are successful, the pipeline pushes the code back to the mainframe via [AWS CodeDeploy](https://aws.amazon.com/cn/codedeploy/) and an intermediary server. On the mainframe side, the code can go through a recompilation and final testing before being pushed to production.\n\nYou can further optimize operations and licensing cost of mainframe emulator by leveraging the managed integrated development and test environment provided by [AWS Mainframe Modernization](https://aws.amazon.com/cn/mainframe-modernization/) service.\n\n\n\nFigure 2. Mainframe CI/CD augmentation\n\n\n### **Pattern 3: Augment mainframe with agile data analytics on AWS**\n\n\nCore business applications running on mainframes generate a lot of data throughout the years. Decades of historical business transactions and massive amounts of user data present an opportunity to develop deep business insight. By creating a data lake using the AWS big data services, you can gain faster analytics capabilities and better insight into core business data originated from mainframe applications.\n\nFigure 3 depicts data being pulled from relational, hierarchical, or mainframe file-based data stores on mainframes. These data are presented in various formats and stored as DB2 for z/OS, VSAM, IMS DB, IDMS, DMS, or other formats. You can use AWS partners data replication and change [data capture tools](https://docs.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server?view=sql-server-ver16) from [AWS Marketplace](https://aws.amazon.com/marketplace/) or AWS cloud services, such as [Amazon Managed Streaming for Apache Kafka](https://aws.amazon.com/cn/msk/) for near real-time data streaming, Transfer Family services, and DataSync for moving data in batch from mainframes to AWS.\n\nOnce data are replicated to AWS, you can further process data using the services like [AWS Lambda](https://aws.amazon.com/cn/lambda/), or [Amazon Elastic Container Service](https://aws.amazon.com/cn/ecs/) and store the processed data on various AWS storage services, such as [Amazon DynamoDB](https://aws.amazon.com/cn/dynamodb/), [Amazon Relational Database Service](https://aws.amazon.com/rds/), and Amazon S3.\n\nBy using AWS big data and data analytics services, such as [Amazon EMR](https://aws.amazon.com/emr/), [Amazon Redshift](https://aws.amazon.com/redshift/),[ Amazon Athena](https://aws.amazon.com/athena/), [AWS Glue](https://aws.amazon.com/glue/), and [Amazon QuickSight](https://aws.amazon.com/quicksight/), you can develop deep business insight and present flexible visuals to your customers. [Read more about mainframe data integration](https://aws.amazon.com/blogs/architecture/mainframe-offloading-and-modernization-using-mainframe-data-to-build-cloud-native-services-with-aws/).\n\n\n\nFigure 3. Mainframe data analytics augmentation\n\n\n### **Pattern 4: Augment mainframe with new functions and channels on AWS**\n\n\nOrganizations with a mainframe use AWS to innovate and iterate quickly, as they often lack agility. For example, a common scenario for a bank could be providing a mobile application for customer engagements, such as supporting a marketing campaign for a new credit card.\n\nAs depicted in Figure 4, with the data replicated from mainframes to AWS cloud and analyzed by AWS big data and analytics services, new business functions can be developed on cloud-native applications by using [Amazon API Gateway](https://aws.amazon.com/api-gateway/), AWS Lambda, and [AWS Fargate](https://aws.amazon.com/fargate/). These new business applications can interact with mainframe data, and the combination can give deep business insight.\n\nTo add new innovation capabilities, with time-series data generated by the new business function applications, using [Amazon Forecast](https://aws.amazon.com/forecast/) can predict domain-specific metrics, such as inventory, workforce, web traffic, and finances. [Amazon Lex ](https://aws.amazon.com/lex/)can build virtual agents, automate informational response to customer enquiries, and improve business productivity. Adding [Amazon SageMaker](https://aws.amazon.com/sagemaker/), you can prepare data gathered from mainframe and new business applications at scale to build, train, and deploy machine learning models for any business cases.\n\nYou can further improve customer engagement by incorporating [Amazon Connect ](https://aws.amazon.com/connect/)and [Amazon Pinpoint](https://aws.amazon.com/pinpoint/) to build multichannel communications.\n\n\n\nFigure 4. Mainframe new functions and channels augmentation\n\n\n### **Conclusion**\n\n\nTo increase agility, maximize the value of investments, and innovate faster, organizations can adopt the patterns discussed in this post to augment mainframes by using AWS services to build resilient data protection solution, provision agile CI/CD integrated development and test environment, liberate mainframe data and developing innovation solutions for new digital customer experience. With AWS Mainframe Modernization service, you can accelerate this journey and innovate faster.\n\n\n\n### **Lewis Tang**\n\nLewis Tang is a Senior Solutions Architect at Amazon Web Services based in Sydney, Australia. Lewis provides partners guidance to a broad range of AWS services and help partners to accelerate AWS practice growth.","render":"<p>Customers with mainframes want to use Amazon Web Services (AWS) to increase agility, maximize the value of their investments, and innovate faster. On June 8, 2022, AWS announced the general availability of AWS Mainframe Modernization, a new service that makes it faster and simpler for customers to modernize mainframe-based workloads.</p>\n<p>In this post, we discuss the common use cases and the augmentation architecture patterns that help liberate data from mainframe for modern data analytics, get rid of expensive and unsupported tape storage solutions for mainframe, build new capabilities that integrate with core mainframe workloads, and enable agile development and testing by adopting CI/CD for mainframe.</p>\n<h3><a id=\"Pattern_1_Augment_mainframe_data_retention_with_backup_and_archival_on_AWS_5\"></a><strong>Pattern 1: Augment mainframe data retention with backup and archival on AWS</strong></h3>\n<p>Mainframes process and generate the most business-critical data. It’s imperative to provide data protection via solutions, such as data backup, archiving, and disaster recovery. Mainframes usually use automated tape libraries—virtual tape libraries for backup and archive. These tapes need to be stored, organized, and transported to vaults and disaster recovery sites. All this can be very expensive and rigid.</p>\n<p>There is a more cost-effective approach that helps simplify the operations of tape libraries: leverage AWS partner tools, such as <a href=\"https://aws.amazon.com/cn/blogs/apn/how-cloud-backup-for-mainframes-cuts-costs-with-model9-and-aws/\" target=\"_blank\">Model9</a>, to transparently migrate the data on tape storage to AWS.</p>\n<p>As depicted in Figure 1, mainframe data can be transferred via the secured network connection using <a href=\"https://aws.amazon.com/cn/aws-transfer-family/\" target=\"_blank\">AWS Transfer Family</a> services or <a href=\"https://aws.amazon.com/cn/datasync/\" target=\"_blank\">AWS DataSync</a> to AWS cloud storage services, such as <a href=\"https://aws.amazon.com/cn/efs/\" target=\"_blank\">Amazon Elastic File System</a>, <a href=\"https://aws.amazon.com/cn/ebs/\" target=\"_blank\">Amazon Elastic Block Store</a>, and <a href=\"https://aws.amazon.com/s3/\" target=\"_blank\">Amazon Simple Storage Service (S3)</a>. After data is stored in AWS cloud, you can configure and move data among these services to meet with the business data processing need. Depending on data storage requirements, data storage costs can be further optimized by configuring <a href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html\" target=\"_blank\">S3 Lifecyle</a> policies to move data among <a href=\"https://aws.amazon.com/cn/s3/storage-classes/\" target=\"_blank\">Amazon S3 storage classes</a>. For long-term data archiving purpose, you can choose <a href=\"https://aws.amazon.com/cn/s3/storage-classes/glacier/\" target=\"_blank\">S3 Glacier storage class</a> to achieve durability, resilience, and the optimal cost effectiveness.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/470a1982bfde4fb1af11cfce83b3586c_image.png\" alt=\"image.png\" /></p>\n<p>Figure 1. Mainframe data backup and archival augmentation</p>\n<h3><a id=\"Pattern_2_Augment_mainframe_with_agile_development_and_test_environments_including_CICD_pipeline_on_AWS_19\"></a><strong>Pattern 2: Augment mainframe with agile development and test environments including CI/CD pipeline on AWS</strong></h3>\n<p>For any business-critical business application, a typical mainframe workload requires development and test environments to support production workloads. It’s common to see the lengthy application development lifecycle, a lack of automated testing, and an absent CI/CD pipeline with most of mainframes. Furthermore, the existing mainframe development processes and tools are outdated, as they are unable to keep up with the business pace, resulting in a growing backlog. Organizations with mainframes look for application development solutions to solve these challenges.</p>\n<p>As demonstrated in Figure 2, AWS developer tools orchestrate code compilation, testing, and deployment among mainframe test environments. Mainframe test environments are either provided by the mainframe vendors as emulators or by AWS partners, such as <a href=\"https://www.microfocus.com/en-us/home\" target=\"_blank\">Micro Focus</a>. You can load the preferred developer tools and run an integrated development environment (IDE) from [Amazon WorkSpaces](Amazon WorkSpaces) or <a href=\"https://aws.amazon.com/appstream2/\" target=\"_blank\">Amazon AppStream 2.0</a>. Developers create or modify code in the IDE, and then commit and push their code to <a href=\"https://aws.amazon.com/cn/codecommit/\" target=\"_blank\">AWS CodeCommit</a>. As soon as the code is pushed, an event is generated and triggers the pipeline in <a href=\"https://aws.amazon.com/cn/codepipeline/\" target=\"_blank\">AWS CodePipeline</a> to build the new code in a compilation environment via <a href=\"https://aws.amazon.com/cn/codebuild/\" target=\"_blank\">AWS CodeBuild</a>. The pipeline pushes the new code to the test environment.</p>\n<p>To optimize cost, you can scale the test environment capacity to meet needs. The tests are executed, and the test environment can be shut down when not in use. When the tests are successful, the pipeline pushes the code back to the mainframe via <a href=\"https://aws.amazon.com/cn/codedeploy/\" target=\"_blank\">AWS CodeDeploy</a> and an intermediary server. On the mainframe side, the code can go through a recompilation and final testing before being pushed to production.</p>\n<p>You can further optimize operations and licensing cost of mainframe emulator by leveraging the managed integrated development and test environment provided by <a href=\"https://aws.amazon.com/cn/mainframe-modernization/\" target=\"_blank\">AWS Mainframe Modernization</a> service.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/f7bdcc0359574992b64a1a157a46ef44_image.png\" alt=\"image.png\" /></p>\n<p>Figure 2. Mainframe CI/CD augmentation</p>\n<h3><a id=\"Pattern_3_Augment_mainframe_with_agile_data_analytics_on_AWS_35\"></a><strong>Pattern 3: Augment mainframe with agile data analytics on AWS</strong></h3>\n<p>Core business applications running on mainframes generate a lot of data throughout the years. Decades of historical business transactions and massive amounts of user data present an opportunity to develop deep business insight. By creating a data lake using the AWS big data services, you can gain faster analytics capabilities and better insight into core business data originated from mainframe applications.</p>\n<p>Figure 3 depicts data being pulled from relational, hierarchical, or mainframe file-based data stores on mainframes. These data are presented in various formats and stored as DB2 for z/OS, VSAM, IMS DB, IDMS, DMS, or other formats. You can use AWS partners data replication and change <a href=\"https://docs.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server?view=sql-server-ver16\" target=\"_blank\">data capture tools</a> from <a href=\"https://aws.amazon.com/marketplace/\" target=\"_blank\">AWS Marketplace</a> or AWS cloud services, such as <a href=\"https://aws.amazon.com/cn/msk/\" target=\"_blank\">Amazon Managed Streaming for Apache Kafka</a> for near real-time data streaming, Transfer Family services, and DataSync for moving data in batch from mainframes to AWS.</p>\n<p>Once data are replicated to AWS, you can further process data using the services like <a href=\"https://aws.amazon.com/cn/lambda/\" target=\"_blank\">AWS Lambda</a>, or <a href=\"https://aws.amazon.com/cn/ecs/\" target=\"_blank\">Amazon Elastic Container Service</a> and store the processed data on various AWS storage services, such as <a href=\"https://aws.amazon.com/cn/dynamodb/\" target=\"_blank\">Amazon DynamoDB</a>, <a href=\"https://aws.amazon.com/rds/\" target=\"_blank\">Amazon Relational Database Service</a>, and Amazon S3.</p>\n<p>By using AWS big data and data analytics services, such as <a href=\"https://aws.amazon.com/emr/\" target=\"_blank\">Amazon EMR</a>, <a href=\"https://aws.amazon.com/redshift/\" target=\"_blank\">Amazon Redshift</a>,<a href=\"https://aws.amazon.com/athena/\" target=\"_blank\"> Amazon Athena</a>, <a href=\"https://aws.amazon.com/glue/\" target=\"_blank\">AWS Glue</a>, and <a href=\"https://aws.amazon.com/quicksight/\" target=\"_blank\">Amazon QuickSight</a>, you can develop deep business insight and present flexible visuals to your customers. <a href=\"https://aws.amazon.com/blogs/architecture/mainframe-offloading-and-modernization-using-mainframe-data-to-build-cloud-native-services-with-aws/\" target=\"_blank\">Read more about mainframe data integration</a>.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/9906ffa184694a479c93426ee1f1cc42_image.png\" alt=\"image.png\" /></p>\n<p>Figure 3. Mainframe data analytics augmentation</p>\n<h3><a id=\"Pattern_4_Augment_mainframe_with_new_functions_and_channels_on_AWS_51\"></a><strong>Pattern 4: Augment mainframe with new functions and channels on AWS</strong></h3>\n<p>Organizations with a mainframe use AWS to innovate and iterate quickly, as they often lack agility. For example, a common scenario for a bank could be providing a mobile application for customer engagements, such as supporting a marketing campaign for a new credit card.</p>\n<p>As depicted in Figure 4, with the data replicated from mainframes to AWS cloud and analyzed by AWS big data and analytics services, new business functions can be developed on cloud-native applications by using <a href=\"https://aws.amazon.com/api-gateway/\" target=\"_blank\">Amazon API Gateway</a>, AWS Lambda, and <a href=\"https://aws.amazon.com/fargate/\" target=\"_blank\">AWS Fargate</a>. These new business applications can interact with mainframe data, and the combination can give deep business insight.</p>\n<p>To add new innovation capabilities, with time-series data generated by the new business function applications, using <a href=\"https://aws.amazon.com/forecast/\" target=\"_blank\">Amazon Forecast</a> can predict domain-specific metrics, such as inventory, workforce, web traffic, and finances. <a href=\"https://aws.amazon.com/lex/\" target=\"_blank\">Amazon Lex </a>can build virtual agents, automate informational response to customer enquiries, and improve business productivity. Adding <a href=\"https://aws.amazon.com/sagemaker/\" target=\"_blank\">Amazon SageMaker</a>, you can prepare data gathered from mainframe and new business applications at scale to build, train, and deploy machine learning models for any business cases.</p>\n<p>You can further improve customer engagement by incorporating <a href=\"https://aws.amazon.com/connect/\" target=\"_blank\">Amazon Connect </a>and <a href=\"https://aws.amazon.com/pinpoint/\" target=\"_blank\">Amazon Pinpoint</a> to build multichannel communications.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/381af9ad0e434752a437bc11f991f0fa_image.png\" alt=\"image.png\" /></p>\n<p>Figure 4. Mainframe new functions and channels augmentation</p>\n<h3><a id=\"Conclusion_67\"></a><strong>Conclusion</strong></h3>\n<p>To increase agility, maximize the value of investments, and innovate faster, organizations can adopt the patterns discussed in this post to augment mainframes by using AWS services to build resilient data protection solution, provision agile CI/CD integrated development and test environment, liberate mainframe data and developing innovation solutions for new digital customer experience. With AWS Mainframe Modernization service, you can accelerate this journey and innovate faster.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/6d9bb3c204494c8c82bdb85f977a7975_image.png\" alt=\"image.png\" /></p>\n<h3><a id=\"Lewis_Tang_74\"></a><strong>Lewis Tang</strong></h3>\n<p>Lewis Tang is a Senior Solutions Architect at Amazon Web Services based in Sydney, Australia. Lewis provides partners guidance to a broad range of AWS services and help partners to accelerate AWS practice growth.</p>\n"}