Job Title – Data Engineer
Qualification – B.E/B.Tech/BCA/MCA/MSc
Company Name – IBM
Location – Bangalore
Salary – Rs. 3 to 5 L.P.A.
Job Info.:-
Developed Python and pyspark programs for data analysis.. Good working experience with python to develop Custom Framework for generating of rules (just like rules engine).
Developed Hadoop streaming Jobs using python for integrating python API supported applications..
Developed Python code to gather the data from HBase and designs the solution to implement using Pyspark.
Preferred Technical and Professional Expertise-
Apache Spark DataFrames/RDD’s were used to apply business transformations and utilized Hive Context objects to perform read/write operations..
Re – write some Hive queries to Spark SQL to reduce the overall batch time.