Закрыт

Data Processing Software Architecture

Purpose

This project is a proof of concept of a modern data processing engine that will bring to ingest, explore, prepare, manage, and serve data for immediate BI and machine learning needs. Currently, it is restricted to personal usage only.

Scope

The purpose of the modern data processing engine is to bring modern data warehousing capabilities with reasonable pricing ease.

System Features

• At the moment, the system is powered by a spark engine and other components.

• Functional Requirements: The system connects to the following data sources.

 MS SQL Server

 AWS Redshift

 Azure synapse

 web APIs

 Flat Files

 Snowflake

• The system has query engine capacity using SQL which allows running federated queries on the above-mentioned data sources except for web API.

• The system has integrated data processing functionality that allows users to use Apache Spark to process data from the above-mentioned data sources.

• The system has directed acyclic graph capability using Air flow.

• The system provides a search facility that will allow full text searching of all data in the memory of the engine that the user is permitted to access. The system must support the following searches:

 find all words specified.

 find any word specified.

 find the exact phrase.

• The software is built on a framework or architect which lets it run on distributed systems.

• Secure impersonation inside its execution model for added security and offers data encryption and masking services.

• The system has connectors to share the output to third-party BI tools such as Power BI, or Looker.

Job Requirements

• Experienced architecting and developing software for scalable, distributed systems.

• Understanding the current architecture of the application and assisting in scaling the architecture cost effectively.

• Develop orchestration in the application.

• Enable users to schedule jobs on the basis of time and event.

• Containerization/packaging of the application so the application is not reliant on any local systems.

• Cost analysis of current configuration & architecture of the application.

• Comparative analysis of applications with various configuration with the same architecture.

• Writing efficient and modular code.

• Experience with cloud technologies and distributed system servers.

• Ability to facilitate demonstrations, proof of concepts.

• Deep understanding of spark application.

• Able to expertly convey ideas and concepts to others.

• Understanding of the public cloud market and pain points driving enterprise cloud adoption.

• In-depth understanding and the ability to demonstrate expertise in designing, deploying, and maintaining custom enterprise web applications.

• Prepare a high-level PowerPoint presentation and detailed word document of the application on completion of the project.

Nice to Have

• Strong knowledge of Python Machine Learning standard libraries.

• Strong understanding of all commonly used Machine Learning models and the main algorithms that compose the models.

• Good understanding of the built-in data types. (Lists, dictionaries, tuples sets).

Навыки: Разработка баз данных, Обработка данных, Архитектура ПО

Показать больше: form collecting data processing software, data processing flow concept data entry, access data processing software, project data processing analysis writting reort software spss, act data processing software, data processing comparison details software buy, data processing software house hyderabad sindh, software architecture data security related projects, data processing template software, mobile/tablet data collection software needed like surveytogo with offline and online data collection purpose, mobiletablet data collection software needed like surveytogo with offline and online data collection purpose, online concept of data processing, electronic data processing software, jullious software company for outsourced data processing, am new member in this but i have 9 years experience in online processing and data processing also i have don data entry project, lidar data processing software, data processing software, data processing software list, what is data processing software, different between word processing and data processing software

О работодателе:
( 0 отзыв(-а, -ов) ) Brampton, Canada

ID проекта: #32246717

17 фрилансеров(-а) готовы выполнить эту работу в среднем за $2562

developer2581

Hi, This is M Zahid I went through the details of your project and would like to offer our services to deliver the results that you expect and we are sure you will not be disappointed. I got interested in your proje Больше

$2600 CAD за 7 дней(-я)
(29 отзывов(-а))
6.9
fastwebsoft

Hello, Hope you’re fit and fine during this pandemic era! We have gone through the requirements of yours and love to say that we have already created many customized software solutions for my clients. In fact not onl Больше

$2600 CAD за 20 дней(-я)
(13 отзывов(-а))
6.0
tecogno

Hi, We at Tecogno Solutions are a team of Passionate Data Science and Full Stack professionals having more than five years of combined experience in multiple areas including Backend, Frontend, Machine learning (ML), C Больше

$3000 CAD за 7 дней(-я)
(1 отзыв)
4.7
(2 отзывов(-а))
3.8
hussainmuzammal4

This project is a proof of concept of a modern data processing engine that will bring to ingest, explore, prepare, manage, and serve data for immediate BI and machine learning needs. Currently, it is restricted to pers Больше

$2600 CAD за 7 дней(-я)
(3 отзывов(-а))
3.2
montylc

Hello, Greetings! ___Hope you're reading my proposal, will wait for your positive response.___ ___I'm interested in your project.___ ___I can start the work right away & I assure you for the best quality work.___ I'v Больше

$2600 CAD за 7 дней(-я)
(1 отзыв)
3.2
muhaamadmaaz

Hi, I have very a good understanding of data processing systems, in fact I developed an application which allowed to gather data from 1 million shopify stores, which was used to later to apply machine learning usage. Больше

$1950 CAD за 7 дней(-я)
(5 отзывов(-а))
3.2
AmolZinjadeP

Hi, I’m Senior Data Architect working for Globant company with having many Hadoop ecosystems like HDFS, Spark, Hive, ETL, Azure etc Please contact me for more details

$3000 CAD за 7 дней(-я)
(5 отзывов(-а))
2.8
Pankaj9810

Hey there ! Good day. I am Pankaj Ranga I would like to work with your project and I can surely provide the details you have specified on the project description. I can confidently say that I am knowledgeable enough to Больше

$2600 CAD за 7 дней(-я)
(2 отзывов(-а))
3.0
Slmmy

Hi There, I am flexible with my working hours and would appreciate it if you could discuss your project as soon as possible. I would greatly appreciate the opportunity to be working with you and to discuss my qualifi Больше

$2600 CAD за 7 дней(-я)
(1 отзыв)
2.0
imrajkumar94

Hello Greetings for the day, Am a professional freelancer expertized in various MSBI ,Power platform, Python and Azure data tools , with hands-on experience around 5 years in different segments of business, I will pro Больше

$2600 CAD за 7 дней(-я)
(3 отзывов(-а))
1.6
jitendra2013

Hi there, I am a Senior Software Engineer, having 9+ years of top experience in .NET Web, SaaS, Database, Migration, API, Library, Services, Reports, Dashboards. I can analyze, design, develop and manage small to larg Больше

$3000 CAD за 30 дней(-я)
(2 отзывов(-а))
3.4
nayeepiyush3

Hello I will do and i am not zero profile my old profile has bin close and i have a Big Company my company Name Webblu Technologies [login to view URL] I am read your project description carefully, and i am expert and fastly wor Больше

$2600 CAD за 36 дней(-я)
(0 отзывов(-а))
0.0
nishitpatelsoft

Hello there, Hope you are doing well! I have 5+ years of strong experience as Python Developer. I have expertise with Django, Flask, Python, AI, ML, NLP and Data Science. Can we have a quick chat/call to discuss thi Больше

$2600 CAD за 7 дней(-я)
(0 отзывов(-а))
0.0
Purohit18

Hi, Thanks for the detailed requirements We have a hands-on experience in the ML Domain and we can develop this software for you. Here is one of our ML-based project: -[login to view URL] I can share my portfo Больше

$2600 CAD за 7 дней(-я)
(0 отзывов(-а))
0.0
andrewdai11191

Hello Thanks for your job posting I read your job description carefully I am an Python expert I have a lots of experience in Data Processing If you work with me, you will get best good results Please give me your detai Больше

$2000 CAD за 20 дней(-я)
(0 отзывов(-а))
0.0
vinod00k

Hello there , i am a full stack software developer having 5+ years experience , i have experience in developing data processing softwares , database softwares, APK softwares , Mobile apps and website , i have complet Больше

$2000 CAD за 7 дней(-я)
(0 отзывов(-а))
0.0