Skip to main content

Posts

Showing posts from October 24, 2024

Pyspark load and transform guidewire table with it's datatype autodetecting capacity

Continue from blog post -->  Guidewire Self Managed version H2 Tables ingest into Databricks Delta Tables Published Notebook for PySpark will load Guidewire CSV files and then create delta tables based on auto-detecting datatypes and then finally all the load and transformation Process also creating in separate delta file processing log table final steps Here is the Code available on databricks Create New Guidewire Claim Tables from CSV File Auto-Detect Data Type V1.0 Template Code logic below  1. Create New Guidewire Claim Tables from CSV File Auto-Detect Data Type V1.0 Template from pyspark.sql import SparkSession from pyspark.sql.types import StringType, IntegerType, FloatType, BooleanType, TimestampType from pyspark.sql.utils import AnalysisException # Initialize Spark session (already initialized in Databricks) spark = SparkSession.builder.getOrCreate() # Define the new database name new_database = "gw_claimcenter_raw_db_2" # Step 1: Create a new database if it ...