High Performance Computing (HPC) in cloud

Jul 7, 2017Download as pptx, pdf2 likes1,832 views

Learn from Accubits Technologies High Performance Computing (HPC) most generally refers to the practice of aggregating computing power in a way that delivers much higher performance than one could get out of a typical desktop computer or workstation in order to solve large problems in science, engineering, or business.

High Performance Computing (HPC) in cloud

Overview
1. Introduction to HPC
2. Hadoop (HDFS, MapReduce)
3. AWS toolkit (Amazon S3, Amazon EMR, Amazon Redshift)
4. Case study

Large data files from sequencers.
Computational bottleneck.
Processing time.
Data persistence and reliability.
Data security.
Bottlenecks in Genome Analysis

Introduction
“High Performance Computing (HPC) most generally refers to the practice of
aggregating computing power in a way that delivers much higher performance
than one could get out of a typical desktop computer or workstation in order to
solve large problems in science, engineering, or business. ”

Dedicated supercomputer.
Commodity HPC cluster.
Grid computing.
HPC in cloud.
Forms of HPC

Hadoop
Open source Java based framework for reliable, scalable and distributed
computing.
Doug Cutting and Mike Cafarella, 2006-08 in Yahoo!- inspired by Google (GFS)
in 2003.
Key Components
Hadoop Distributed File System (HDFS)
MapReduce

Hadoop
HDFS (Hadoop Distributed File System)
Data management layer
Master-Slave architecture
Fault Tolerant
Key Components:
NameNode
SecondaryNamenode

Hadoop- Continued
MapReduce
Mappers and Reducers
Batch oriented
Key Components
JobTracker
TaskTracker

AWS ToolKit - Amazon Elastic MapReduce (EMR)
Managed Hadoop framework.
Runs almost all popular distributed frameworks such as Apache Spark, HBase,
Presto, and Flink.
Elastic.
Flexible Data storage (S3, HDFS, RedShift, Glacier, RDS).
Secure and reliable.
Full control and root access.

AWS ToolKit - Amazon EMR
aws emr create-cluster
--name "demo"
--release-label emr-4.5.0
--instance-type m3.xlarge
--instance-count 2
--ec2-attributes KeyName=YOUR-AWS-SSH-KEY
--use-default-roles
--applications Name=Hive Name=Spark

aws emr create-cluster
--name "Test cluster"
--ami-version 2.4
--applications Name=Hive Name=Pig
--use-default-roles --ec2-attributes KeyName=myKey
--instance-groups
InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge
InstanceGroupType=CORE,InstanceCount=2,InstanceType=m3.xlarge
--steps Type=PIG,Name="Pig Program",ActionOnFailure=CONTINUE,
Args=[-f,s3://mybucket/scripts/pigscript.pig,-p,
INPUT=s3://mybucket/inputdata/,-p,
OUTPUT=s3://mybucket/outputdata/,
$INPUT=s3://mybucket/inputdata/,
$OUTPUT=s3://mybucket/outputdata/]

AWS ToolKit - Amazon S3 (Simple Storage Service)
Virtually infinite storage.
Single object size up to 5TB.
Why use S3?
Durable, Low Cost, Scalable, High Performance, Secure, Integrated, Easy to Use.
Decouple storage and computation resources.
HDFS requirements and implements EMRFS.

AWS ToolKit - Amazon Redshift
Fast, simple petabyte-scale data warehouse.
Use SQL query to interact.
Massively parallel.
Relational.
Architecture - Leader Node and Compute node.
Fast - 4 GB/sec/node.

Case Study- Rail RNA
Cloud-enabled spliced aligner that analyzes many samples at once.
Architecture - Amazon S3, Amazon EMR.
~50000 (from NCBI archive) human RNA sample using Rail-RNA - 150 Tbps.
Input to result - 2 weeks.
Cost- ~$1.40/sample.
Paper- Splicing across SRA.

This document provides an overview and syllabus for an AIX System Administration class that will take place over 5 days from 9:30am to 5:30pm. The class will cover topics such as Unix and AIX overviews, IBM POWER servers, installing the AIX operating system, and logging into the system. Hands-on experience will be provided through virtualized AIX systems on IBM POWER7 blades in the classroom lab network.

SDWAN.pdfsushil kumar

This document discusses the benefits of SD-WAN and how it addresses challenges businesses face with their networks. Some key points: - SD-WAN allows businesses to use cheaper internet connections instead of expensive private circuits, reducing costs while maintaining security and performance. It provides dynamic routing of traffic over multiple links for high availability. - SD-WAN simplifies management through centralized, cloud-based control. It improves application performance by selecting optimal paths and dynamically moving traffic if there are issues. - SD-WAN enhances security with features like firewalls, intrusion detection, and encryption. Management has visibility into network and application usage for planning. - Case studies show how SD-WAN helped

Traffic Engineering Using Segment Routing Cisco Canada

1) The document discusses using segment routing for traffic engineering. It provides an overview of segment routing technology, use cases, control and data plane operations, and how segment routing can be used for traffic engineering. 2) Key aspects covered include how segment routing works by encoding a path as an ordered list of segments, different types of segments (IGP prefixes, adjacencies, BGP), and how this allows for application-engineered end-to-end paths. 3) Traffic engineering with segment routing provides explicit routing, supports constraint-based routing without needing RSVP-TE, and uses existing IGP extensions to advertise link attributes.

Presentation CentOS rommel gavia

CentOS is a community-developed Linux distribution derived from Red Hat Enterprise Linux source code. It is freely available and aims to provide a long-term enterprise-class OS while removing Red Hat branding and logos. CentOS versions correspond to RHEL versions and architectures. While not affiliated with or supported by Red Hat, the CentOS community provides support forums and its business model relies on donations and commercial partners.

AI Chip Trends and ForecastCastLabKAIST

- The document discusses trends in AI chips, including the rise of deep learning models enabled by increased computing power and data availability. - It outlines the AI stack from algorithms and neural network models down to chips, memory, and hardware. Popular deep learning model types and applications are also summarized. - The trends are towards more specialized hardware like Google's TPUs for cloud servers and dedicated chips for mobile/edge devices from companies like Qualcomm and Nvidia. Processing-in-memory and new memory technologies may help address bandwidth bottlenecks. - Overall hardware is still catching up to the needs of large neural networks, and there is a lack of unified software tools and frameworks to program diverse AI accelerators.

RedHat LinuxApo

RedHat was the first commercial Linux distribution, created in 1994. It has since grown significantly and is now known as RedHat Enterprise Linux. Some key features of RHEL include virtualization support, optimizations for multi-core processors and network performance, as well as security, directory and packaging features. RHEL provides a flexible, customizable OS for enterprises at an affordable price.

Cisco Data Center Orchestration SolutionCisco Canada

Cisco Data Center Orchestration Solution By: Sasha Lebovic, Data Center Consulting Systems Engineer Join Cisco for a presentation on the management and orchestration of modern virtualized multiservice data center (VMDC) architectures built on integrated compute stack designs. The presentation will provide the audience with an overview of industry directions and Cisco participation in open industry standards such as OpenStack, as well as discussing specific Cisco and partner branded offerings in this area. The presentation will then focus on Cisco’s newest offerings for datacenter automation including the Cisco UCS Director as well BMCs Cloud Lifecycle Manager products. The presentation will include a both a discussion, as well as demonstration of these products capability to provide automated datacenter operations as well as the key services for deploying private and public cloud offerings.

SOC trainingKirill Ermakov

- The document discusses how to properly set up and run a security operations center (SOC). It emphasizes that collecting every event is not useful and outlines some common mistakes made, such as overreliance on templates and ignorance of infrastructure details. - Perfect correlation rules cannot be developed out of the box. Anomaly detection is useless without proper asset, change, and documentation management. The best metric for measuring a SOC's effectiveness is the ratio of registered attacks to actual attacks performed. - Penetration testing, including red team exercises, is recommended to test defenses. Such an exercise revealed issues with the discussed SOC, including only registering about 70% of attacks and vulnerabilities that allowed access to security team laptops. Continuous

3D V-Cache AMD

The document describes AMD's 3D V-Cache technology, which implements a 64MB stacked cache for a 7nm x86-64 CPU. It consists of a 7nm CPU core complex die (CCD) with 32MB L3 cache bonded to a 7nm extended L3 die (L3D) with 64MB of additional L3 cache, bringing the total L3 cache to 96MB. The bonding technique used is hybrid bonding, which offers higher interconnect density and efficiency compared to microbump bonding. This provides a large last-level cache to improve CPU performance with minimal increases to latency and area.

Introduction to virtualizationAhmad Hafeezi

Virtualization allows for the creation of virtual versions of hardware platforms, operating systems, storage and network resources through software. It works by imitating hardware resources through a hypervisor software layer that creates virtual machines with virtual hardware. This allows multiple guest operating systems to run in isolation on a single physical machine. Virtualization provides benefits like reduced costs, increased hardware utilization, easier management and testing across different operating systems. Popular virtualization platforms include VMWare, Hyper-V, KVM, Xen and VirtualBox.

Data centerJwala Chapagai

A data center contains large numbers of servers and networking equipment that support business operations. It provides reliable computing resources, redundant power and networking, and high security. Data centers are classified into tiers based on their redundancy and fault tolerance, with tier 4 being the most fault tolerant. The major goals of data centers are to reduce costs, provide 24/7 support, and allow for expansion flexibility. Data centers require environmental controls, reliable power supplies, fire protection systems, and physical security measures to protect the servers and data. Data centers can be in-house, co-location facilities, or managed by service providers to support a variety of hosting needs for enterprises.

Ibm aixmindmajixtrainings

Andes RISC-V vector extension demystified-tutorialRISC-V International

The document provides an overview of vector technology and the RISC-V Vector extension. It discusses SIMD and vector processor concepts, the evolution of vector instruction sets from Intel MMX/SSE/AVX to the RISC-V Vector specification. It also covers vector register file organization, instruction chaining, applications of vector processing, and challenges in implementing vector architectures. Andes Technology is introducing the AndesCoreTM NX27V processor core which implements the RISC-V Vector extension.

Introduction to linux pptOmi Vichare

Linux is an open-source operating system that originated as a personal project by Linus Torvalds in 1991. It can run on a variety of devices from servers and desktop computers to smartphones. Some key advantages of Linux include low cost, high performance, strong security, and versatility in being able to run on many system types. Popular Linux distributions include Red Hat Enterprise Linux, Debian, Ubuntu, and Mint. The document provides an overview of the history and development of Linux as well as common myths and facts about the operating system.

Linux - Introductions to Linux Operating SystemVibrant Technologies & Computers

Data Center Infrastructure Management Powerpoint Presentation SlidesSlideTeam

Introducing Data Center Infrastructure Management PowerPoint Presentation Slides. You can present key drivers for sustainable infrastructure by using our readily available PPT slide deck. The PowerPoint slide deck helps to illustrate the market size, key funding areas of infrastructure, key technology trends in infrastructure in a presentable manner. Further, the slideshow also showcases the asset management process with lifecycle and framework. Utilize our visually-attention-grabbing asset infrastructure PowerPoint templates to mention inventory assets for manufacturing companies. Showcase the deterioration modeling and the types of deterioration modeling such as asset and risk assessment deterioration modeling with the help of asset infrastructure PPT visual. With this infrastructure management PPT slide deck, explain each deterioration modeling in detail using this PPT slideshow. It is possible to depict asset management decision journey, and performance and cost functions by downloading this PPT presentation. https://bit.ly/3c8cORj

Dell Technologies Dell EMC Data Protection Solutions On One Single Page - POS...Smarter.World

CCNA17 KVM and CloudStackShapeBlue

Safety-Certifying Open Source Software: The Case of the Xen HypervisorStefano Stabellini

Safety is important to software everywhere human lives are at risk. In these environments often safety-certifications are required to ensure that the quality of the software is high enough to minimize the risk of harm to humans. Safety-certifications such as ISO 26262 come with a series of requirements and processes that sometimes clash with well-established Open Source software development practices. How do we reconcile safety-certifications with Open Source? This presentation will provide an answer to that question. Taking Xen as an example of an Open Source project with a rich 15+ years history, this presentation will explain the best way to match Open Source activities with safety-certification requirements. It will discuss the role of the upstream community and downstream vendors in achieving compliance with ISO 26262 and IEC 61508. It will go through the changes to Xen Project processes already underway and the ones planned for the future to align the Xen hypervisor with safety-certifications. The talk will cover MISRA, traceability, testing, etc., and the latest updates from the Xen FuSa working group.

kali linux.pptxanumeha bhatnagar

Kali Linux is a Debian-derived Linux distribution designed for digital forensics and penetration testing. It contains over 600 security tools for tasks like reconnaissance, vulnerability assessment, password attacks, and wireless attacks. It is pre-installed with tools like Nmap, Wireshark, Aircrack-ng, Hydra, and Burpsuite. Kali Linux can also be installed on Android devices through tools like Linux Deploy to enable penetration testing on mobile networks and devices.

ASA Firepower NGFW Update and Deployment ScenariosCisco Canada

This session will focus on typical deployment scenarios for the Adaptive Security Appliance family running FirePower Services. Also, a feature overview and comparison of the ASA with Firepower services and the new Firepower Threat Defense (FTD) image will be included with updates on the new Firepower hardware platform. Deployment use cases will include Internet Edge, various segmentation scenarios, and VPN. A configuration walk-through and accepted best practices will be covered. This session is designed for existing ASA customers and targets the security and network engineer. They will learn the benefit of a FirePower NGFW in network edge and Internet use cases

VMware NSX 101: What, Why & HowAniekan Akpaffiong

OpenStack Cinder Best Practices - Meet UpAaron Delp

OpenStack Block Storage (Cinder) provides on-demand, self-service access to block storage resources through abstraction and automation of backend storage devices. Cinder uses a plugin architecture that supports various storage backends like LVM, SolidFire, EMC, etc. It allows users to dynamically create, attach, and detach disk volumes to Nova instances. The presentation discusses Cinder's architecture, common storage types, and demos creating volume types and extra specs to control which backend is used.

cisco csr1000vMing914298

This document provides an overview of extending enterprise networks into public clouds using Cisco Cloud Services Router 1000v (CSR1000v). It discusses the state of public clouds and networking challenges in AWS, Azure and GCP. It then introduces the CSR1000v and Cisco solutions for connecting to different public clouds. The presentation agenda includes demos and best practices. Key terms related to networking in public clouds are also defined.

Cisco data center training for ibmChristian Silva Espinoza

This document provides an overview and summary of Cisco's Data Center networking and storage solutions, with a focus on the new Cisco MDS 9710 Director. Some key points: - Cisco offers a multi-protocol portfolio including Fibre Channel, FCoE, and IP networking solutions to address growing data and connectivity demands in modern data centers. - The Cisco MDS 9710 is the newest storage director that provides the highest scalability, availability, and investment protection in the industry for large scale data centers. - It supports up to 384 line-rate 16Gbps Fibre Channel ports or 48-port 10GbE FCoE modules in a single chassis. This provides 3 times the performance of competing

IT Infrastructure Automation with AnsibleDio Pratama

This document discusses automating IT infrastructure using Ansible. It provides an overview of Ansible and how it can be used to manage devices programmatically through infrastructure as code. Specific examples are given of how Ansible can be used to automate tasks like server deployment, configuration changes, backups and application deployments. The benefits of automation through tools like Ansible are reducing repetitive manual tasks and enabling infrastructure to self-heal through integration with monitoring systems.

AMD Chiplet Architecture for High-Performance Server and Desktop ProductsAMD

This document discusses AMD's chiplet architecture for high-performance server and desktop processors. Key points include: - AMD partitions the system-on-a-chip design, using 7nm technology for CPU cores while leaving I/O interfaces in older process nodes. This improves performance and lowers costs. - CPU dies ("chiplets") are connected using high-speed SerDes links both on-package and between dies. This allows for more chiplets and cores than traditional monolithic designs. - Innovations in packaging, power distribution, and operating system scheduling were required to enable the multi-chiplet design and improve performance.

Serverless ArchitectureElana Krasner

Streamline Hadoop DevOps with Apache AmbariAlejandro Fernandez

In Memory Analytics with Apache SparkVenkata Naga Ravi

More Related Content

What's hot (20)

3D V-Cache AMD

Introduction to virtualizationAhmad Hafeezi

Data centerJwala Chapagai

Ibm aixmindmajixtrainings

Andes RISC-V vector extension demystified-tutorialRISC-V International

Introduction to linux pptOmi Vichare

Linux - Introductions to Linux Operating SystemVibrant Technologies & Computers

Data Center Infrastructure Management Powerpoint Presentation SlidesSlideTeam

Dell Technologies Dell EMC Data Protection Solutions On One Single Page - POS...Smarter.World

CCNA17 KVM and CloudStackShapeBlue

Safety-Certifying Open Source Software: The Case of the Xen HypervisorStefano Stabellini

kali linux.pptxanumeha bhatnagar

ASA Firepower NGFW Update and Deployment ScenariosCisco Canada

VMware NSX 101: What, Why & HowAniekan Akpaffiong

OpenStack Cinder Best Practices - Meet UpAaron Delp

cisco csr1000vMing914298

Cisco data center training for ibmChristian Silva Espinoza

IT Infrastructure Automation with AnsibleDio Pratama

AMD Chiplet Architecture for High-Performance Server and Desktop ProductsAMD

Serverless ArchitectureElana Krasner

3D V-Cache AMD

Introduction to virtualizationAhmad Hafeezi

Data centerJwala Chapagai

Ibm aixmindmajixtrainings

Andes RISC-V vector extension demystified-tutorialRISC-V International

Introduction to linux pptOmi Vichare

Linux - Introductions to Linux Operating SystemVibrant Technologies & Computers

Data Center Infrastructure Management Powerpoint Presentation SlidesSlideTeam

Dell Technologies Dell EMC Data Protection Solutions On One Single Page - POS...Smarter.World

CCNA17 KVM and CloudStackShapeBlue

Safety-Certifying Open Source Software: The Case of the Xen HypervisorStefano Stabellini

kali linux.pptxanumeha bhatnagar

ASA Firepower NGFW Update and Deployment ScenariosCisco Canada

VMware NSX 101: What, Why & HowAniekan Akpaffiong

OpenStack Cinder Best Practices - Meet UpAaron Delp

cisco csr1000vMing914298

Cisco data center training for ibmChristian Silva Espinoza

IT Infrastructure Automation with AnsibleDio Pratama

AMD Chiplet Architecture for High-Performance Server and Desktop ProductsAMD

Serverless ArchitectureElana Krasner

Similar to High Performance Computing (HPC) in cloud (17)

Streamline Hadoop DevOps with Apache AmbariAlejandro Fernandez

In Memory Analytics with Apache SparkVenkata Naga Ravi

How containers helped a SaaS startup be developed and go liveRamon Navarro

A Container-based Sizing Framework for Apache Hadoop/Spark ClustersDataWorks Summit/Hadoop Summit

This document proposes a container-based sizing framework for Apache Hadoop/Spark clusters that uses a multi-objective genetic algorithm approach. It emulates container execution on different cloud platforms to optimize configuration parameters for minimizing execution time and deployment cost. The framework uses Docker containers with resource constraints to model cluster performance on various public clouds and instance types. Optimization finds Pareto-optimal configurations balancing time and cost across objectives.

How to run your Hadoop Cluster in 10 minutesVladimir Simek

- Two companies faced challenges processing big data on-premises, including high fixed costs, slow deployment, lack of scalability, and outages impacting production. - Amazon Elastic MapReduce (EMR) provides a managed Hadoop service that allows companies to launch clusters within minutes in the AWS cloud at lower costs by using elastic and scalable infrastructure. - AOL moved their 2PB on-premises Hadoop cluster to EMR, reducing costs by 4x while gaining automatic scaling and high availability across availability zones. EMR addressed their challenges and allowed faster restatement of historical data.

대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016Amazon Web Services Korea

1. The document demonstrates how to use various AWS services like Kinesis, Redshift, Elasticsearch to analyze streaming game log data. 2. It shows setting up an EC2 instance to generate logs, creating a Kinesis stream to ingest the logs, and building Redshift tables to run queries on the logs. 3. The document also explores loading the logs from Kinesis into Elasticsearch for search and linking Kinesis and Redshift with Kinesis Analytics for real-time SQL queries on streams.

Amazon EMR MasterclassIan Massingham

This document provides an overview of Amazon Elastic MapReduce (EMR), including: 1) EMR allows users to quickly and cost-effectively process vast amounts of data by providing a managed Hadoop framework and supporting popular distributed frameworks like Spark. 2) The document demonstrates how to use EMR for tasks like clickstream analysis, log processing, and genomic research through example use cases. 3) It outlines the agenda which will cover Hadoop fundamentals, EMR features, how to get started, supported tools, and additional resources.

Spark,Hadoop,Presto ComparitionSandish Kumar H N

The document discusses several big data frameworks: Spark, Presto, Cloudera Impala, and Apache Hadoop. Spark aims to make data analytics faster by loading data into memory for iterative querying. Presto extends R with distributed parallelism for scalable machine learning and graph algorithms. Hadoop uses MapReduce to distribute computations across large hardware clusters and handles failures automatically. While useful for batch processing, Hadoop has disadvantages for small files and online transactions.

Power Hadoop Cluster with AWS CloudEdureka!

Ess1000 glossaryDeepanshu Gupta

This document provides a glossary of terms related to big data and the MapR Converged Data Platform. It defines over 100 terms across A-Z, including common terms like Apache, Hadoop, MapReduce, and Spark as well as more specialized terms related to MapR technologies like MapR-FS, MapR-DB, and MapR Streams. The glossary is intended to serve as a reference for students taking various MapR Academy courses on big data essentials and the MapR platform.

AWS glue technical enablement trainingInfo Alchemy Corporation

Streamline Hadoop DevOps with Apache AmbariDataWorks Summit/Hadoop Summit

Apache Ambari is an open-source tool for provisioning, managing, and monitoring Hadoop clusters. It allows users to deploy Hadoop clusters, install and manage services, configure settings, and perform rolling upgrades with minimal downtime. Ambari 2.4 includes new features like role-based access control, Grafana integration for visualization, log search capabilities, and improved upgrade workflows.

Streamline Hadoop DevOps with Apache AmbariAlejandro Fernandez

Apache Ambari is an open-source platform for provisioning, managing, and monitoring Hadoop clusters. It provides a single pane of glass for deploying Hadoop clusters, including features for lifecycle management, security, alerting, and visualization of metrics. Ambari 2.4 includes new services, role-based access control, management packs, Grafana integration, and other improvements to simplify Hadoop operations.

App Cap2956v2 121001194956 Phpapp01 (1)outstanding59

1) Hadoop is a framework for distributed processing of large datasets across clusters of computers using a simple programming model. 2) Virtualizing Hadoop enables rapid deployment, high availability, elastic scaling, and consolidation of big data workloads on a common infrastructure. 3) Serengeti is a tool that automates the deployment and management of Hadoop clusters on vSphere in under 30 minutes through simple commands.

App cap2956v2-121001194956-phpapp01 (1)outstanding59

Inside the Hadoop Machine @ VMworldRichard McDougall

The document discusses virtualizing Hadoop clusters on VMware vSphere. It describes how Hadoop enables parallel processing of large datasets across clusters using MapReduce. Virtualizing Hadoop provides benefits like simple operations, high availability, and elastic scaling. The document outlines challenges with using Hadoop and how virtualization addresses them. It provides examples of deploying Hadoop clusters on Serengeti and configuring different distributions. Performance results show little overhead from virtualization and benefits of local storage. Joint engineering with Hortonworks adds high availability to Hadoop master daemons using vSphere features.

Secure Hadoop Cluster With KerberosEdureka!

Streamline Hadoop DevOps with Apache AmbariAlejandro Fernandez

In Memory Analytics with Apache SparkVenkata Naga Ravi

How containers helped a SaaS startup be developed and go liveRamon Navarro

A Container-based Sizing Framework for Apache Hadoop/Spark ClustersDataWorks Summit/Hadoop Summit

How to run your Hadoop Cluster in 10 minutesVladimir Simek

대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016Amazon Web Services Korea

Amazon EMR MasterclassIan Massingham

Spark,Hadoop,Presto ComparitionSandish Kumar H N

Power Hadoop Cluster with AWS CloudEdureka!

Ess1000 glossaryDeepanshu Gupta

AWS glue technical enablement trainingInfo Alchemy Corporation

Streamline Hadoop DevOps with Apache AmbariDataWorks Summit/Hadoop Summit

Streamline Hadoop DevOps with Apache AmbariAlejandro Fernandez

App Cap2956v2 121001194956 Phpapp01 (1)outstanding59

App cap2956v2-121001194956-phpapp01 (1)outstanding59

Inside the Hadoop Machine @ VMworldRichard McDougall

Secure Hadoop Cluster With KerberosEdureka!

More from Accubits Technologies (6)

AI-powered real-time video analytics for Manufacturing sectorAccubits Technologies

AI-powered real-time video analytics for defence sectorAccubits Technologies

Blockchain and IoT For Supply Chain TraceabilityAccubits Technologies

The conventional supply chain is plagued with various issues, most pressing among which is transparency or the lack of it thereof. Being a self-auditing distributed ledger that is accessible to all participants in the network and updated in real time, Blockchain brings in unprecedented levels of transparency to the supply chain. Moreover, its decentralized nature ensures that no single participant along the supply chain holds an unfair advantage or any influence on the data pertaining to the shipment.

ICOs : past, present and futureAccubits Technologies

Blockchain in BioinformaticsAccubits Technologies

Blockchains provide transparency, traceability, and trust through their distributed ledger structure. A transaction recorded in a blockchain is woven permanently into the chain through its unique cryptographic hash. Blockchains can be permissioned, requiring authorization to access, or permissionless with open access. They find many uses including managing genomic data through dividing and encrypting personal information across a network.

Neural Networks - How do they work?Accubits Technologies

This presentation begins with explaining the basic algorithms of machine learning and using the same concepts, discusses in detail 2 supervised learning/deep learning algorithms - Artificial neural nets and Convolutional Neural Nets. The relationship between Artificial neural nets and basic machine learning algorithms such as logistic regression and soft max is also explored. For hands on the implementation of ANN's and CNN's on MNIST dataset is also explained.

AI-powered real-time video analytics for Manufacturing sectorAccubits Technologies

AI-powered real-time video analytics for defence sectorAccubits Technologies

Blockchain and IoT For Supply Chain TraceabilityAccubits Technologies

ICOs : past, present and futureAccubits Technologies

Blockchain in BioinformaticsAccubits Technologies

Neural Networks - How do they work?Accubits Technologies

Recently uploaded (20)

abebaw power point presentation esis october.pptmihretwodage

AWS RDS Presentation to make concepts easy.pptxbharatkumarbhojwani

Chapter 6-3 Introducingthe Concepts .pptxPermissionTafadzwaCh

Introduction to Python_for_machine_learning.pdfgoldenflower34

Introduction to systems thinking tools_Eng.pdfAbdurahmanAbd

2-Raction quotient_١٠٠١٤٦.ppt of physical chemisstrybastakwyry

2024 Digital Equity Accelerator Report.pdfdominikamizerska1

Lagos School of Programming Final Project Updated.pdfbenuju2016

report (maam dona subject).pptxhsgwiswhsAngelPinedaTaguinod

End to End Process Analysis - Cox CommunicationsProcess mining Evangelist

Cox Communications is an American company that provides digital cable television, telecommunications, and home automation services in the United States. Gary Bonneau is a senior manager for product operations at Cox Business (the business side of Cox Communications). Gary has been working in the telecommunications industry for over two decades and — after following the topic for many years — is a bit of a process mining veteran as well. Now, he is putting process mining to use to visualize his own fulfillment processes. The business life cycles are very complex and multiple data sources need to be connected to get the full picture. At camp, Gary shared the dos and don'ts and take-aways of his experience.

TOAE201-Slides-Chapter 4. Sample theoretical basis (1).pdfNhiV747372

AI ------------------------------ W1L2.pptxAyeshaJalil6

This lecture provides a foundational understanding of Artificial Intelligence (AI), exploring its history, core concepts, and real-world applications. Students will learn about intelligent agents, machine learning, neural networks, natural language processing, and robotics. The lecture also covers ethical concerns and the future impact of AI on various industries. Designed for beginners, it uses simple language, engaging examples, and interactive discussions to make AI concepts accessible and exciting. By the end of this lecture, students will have a clear understanding of what AI is, how it works, and where it's headed.

MLOps_with_SageMaker_Template_EN idioma inglésFabianPierrePeaJacob

Day 1 MS Excel Basics #.pptxDay 1 MS Excel Basics #.pptxDay 1 MS Excel Basics...Jayantilal Bhanushali

Dr. Robert Krug - Expert In Artificial IntelligenceDr. Robert Krug

Dr. Robert Krug is a New York-based expert in artificial intelligence, with a Ph.D. in Computer Science from Columbia University. He serves as Chief Data Scientist at DataInnovate Solutions, where his work focuses on applying machine learning models to improve business performance and strengthen cybersecurity measures. With over 15 years of experience, Robert has a track record of delivering impactful results. Away from his professional endeavors, Robert enjoys the strategic thinking of chess and urban photography.

Database administration and management chapter 12saniaafzalf1f2f3

Multi-tenant Data Pipeline OrchestrationRomi Kuntsman

Multi-Tenant Data Pipeline Orchestration — Romi Kuntsman @ DataTLV 2025 In this talk, I unpack what it really means to orchestrate multi-tenant data pipelines at scale — not in theory, but in practice. Whether you're dealing with scientific research, AI/ML workflows, or SaaS infrastructure, you’ve likely encountered the same pitfalls: duplicated logic, growing complexity, and poor observability. This session connects those experiences to principled solutions. Using a playful but insightful "Chips Factory" case study, I show how common data processing needs spiral into orchestration challenges, and how thoughtful design patterns can make the difference. Topics include: Modeling data growth and pipeline scalability Designing parameterized pipelines vs. duplicating logic Understanding temporal and categorical partitioning Building flexible storage hierarchies to reflect logical structure Triggering, monitoring, automating, and backfilling on a per-slice level Real-world tips from pipelines running in research, industry, and production environments This framework-agnostic talk draws from my 15+ years in the field, including work with Airflow, Dagster, Prefect, and more, supporting research and production teams at GSK, Amazon, and beyond. The key takeaway? Engineering excellence isn’t about the tool you use — it’s about how well you structure and observe your system at every level.

Important JavaScript Concepts Every Developer Must Knowyashikanigam1

Mastering JavaScript requires a deep understanding of key concepts like closures, hoisting, promises, async/await, event loop, and prototypal inheritance. These fundamentals are crucial for both frontend and backend development, especially when working with frameworks like React or Node.js. At TutorT Academy, we cover these topics in our live courses for professionals, ensuring hands-on learning through real-world projects. If you're looking to strengthen your programming foundation, our best online professional certificates in full-stack development and system design will help you apply JavaScript concepts effectively and confidently in interviews or production-level applications.

Feature Engineering for Electronic Health Record SystemsProcess mining Evangelist

Oak Ridge National Laboratory (ORNL) is a leading science and technology laboratory under the direction of the Department of Energy. Hilda Klasky is part of the R&D Staff of the Systems Modeling Group in the Computational Sciences & Engineering Division at ORNL. To prepare the data of the radiology process from the Veterans Affairs Corporate Data Warehouse for her process mining analysis, Hilda had to condense and pre-process the data in various ways. Step by step she shows the strategies that have worked for her to simplify the data to the level that was required to be able to analyze the process with domain experts.

Urban models for professional practice 03DanisseLoiDapdap

abebaw power point presentation esis october.pptmihretwodage

AWS RDS Presentation to make concepts easy.pptxbharatkumarbhojwani

Chapter 6-3 Introducingthe Concepts .pptxPermissionTafadzwaCh

Introduction to Python_for_machine_learning.pdfgoldenflower34

Introduction to systems thinking tools_Eng.pdfAbdurahmanAbd

2-Raction quotient_١٠٠١٤٦.ppt of physical chemisstrybastakwyry

2024 Digital Equity Accelerator Report.pdfdominikamizerska1

Lagos School of Programming Final Project Updated.pdfbenuju2016

report (maam dona subject).pptxhsgwiswhsAngelPinedaTaguinod

End to End Process Analysis - Cox CommunicationsProcess mining Evangelist

TOAE201-Slides-Chapter 4. Sample theoretical basis (1).pdfNhiV747372

AI ------------------------------ W1L2.pptxAyeshaJalil6

MLOps_with_SageMaker_Template_EN idioma inglésFabianPierrePeaJacob

Day 1 MS Excel Basics #.pptxDay 1 MS Excel Basics #.pptxDay 1 MS Excel Basics...Jayantilal Bhanushali

Dr. Robert Krug - Expert In Artificial IntelligenceDr. Robert Krug

Database administration and management chapter 12saniaafzalf1f2f3

Multi-tenant Data Pipeline OrchestrationRomi Kuntsman

Important JavaScript Concepts Every Developer Must Knowyashikanigam1

Feature Engineering for Electronic Health Record SystemsProcess mining Evangelist

Urban models for professional practice 03DanisseLoiDapdap

High Performance Computing (HPC) in cloud

2. Overview 1. Introduction to HPC 2. Hadoop (HDFS, MapReduce) 3. AWS toolkit (Amazon S3, Amazon EMR, Amazon Redshift) 4. Case study

3. Why?

4. Large data files from sequencers. Computational bottleneck. Processing time. Data persistence and reliability. Data security. Bottlenecks in Genome Analysis

5. How?

6. Introduction “High Performance Computing (HPC) most generally refers to the practice of aggregating computing power in a way that delivers much higher performance than one could get out of a typical desktop computer or workstation in order to solve large problems in science, engineering, or business. ”

7. Dedicated supercomputer. Commodity HPC cluster. Grid computing. HPC in cloud. Forms of HPC

8. What?

9. Hadoop Open source Java based framework for reliable, scalable and distributed computing. Doug Cutting and Mike Cafarella, 2006-08 in Yahoo!- inspired by Google (GFS) in 2003. Key Components Hadoop Distributed File System (HDFS) MapReduce

10. Hadoop HDFS (Hadoop Distributed File System) Data management layer Master-Slave architecture Fault Tolerant Key Components: NameNode SecondaryNamenode

11. Hadoop- Continued MapReduce Mappers and Reducers Batch oriented Key Components JobTracker TaskTracker

12. Hadoop- Architecture

13. AWS ToolKit - Amazon Elastic MapReduce (EMR) Managed Hadoop framework. Runs almost all popular distributed frameworks such as Apache Spark, HBase, Presto, and Flink. Elastic. Flexible Data storage (S3, HDFS, RedShift, Glacier, RDS). Secure and reliable. Full control and root access.

14. AWS ToolKit - Amazon EMR aws emr create-cluster --name "demo" --release-label emr-4.5.0 --instance-type m3.xlarge --instance-count 2 --ec2-attributes KeyName=YOUR-AWS-SSH-KEY --use-default-roles --applications Name=Hive Name=Spark

15. aws emr create-cluster --name "Test cluster" --ami-version 2.4 --applications Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge InstanceGroupType=CORE,InstanceCount=2,InstanceType=m3.xlarge --steps Type=PIG,Name="Pig Program",ActionOnFailure=CONTINUE, Args=[-f,s3://mybucket/scripts/pigscript.pig,-p, INPUT=s3://mybucket/inputdata/,-p, OUTPUT=s3://mybucket/outputdata/, $INPUT=s3://mybucket/inputdata/, $OUTPUT=s3://mybucket/outputdata/]

16. AWS ToolKit - Amazon S3 (Simple Storage Service) Virtually infinite storage. Single object size up to 5TB. Why use S3? Durable, Low Cost, Scalable, High Performance, Secure, Integrated, Easy to Use. Decouple storage and computation resources. HDFS requirements and implements EMRFS.

17. AWS ToolKit - Amazon Redshift Fast, simple petabyte-scale data warehouse. Use SQL query to interact. Massively parallel. Relational. Architecture - Leader Node and Compute node. Fast - 4 GB/sec/node.

18. Case Study- Rail RNA Cloud-enabled spliced aligner that analyzes many samples at once. Architecture - Amazon S3, Amazon EMR. ~50000 (from NCBI archive) human RNA sample using Rail-RNA - 150 Tbps. Input to result - 2 weeks. Cost- ~$1.40/sample. Paper- Splicing across SRA.

19. Thank you

High Performance Computing (HPC) in cloud

Recommended

More Related Content

What's hot (20)

Similar to High Performance Computing (HPC) in cloud (17)

More from Accubits Technologies (6)

Recently uploaded (20)

High Performance Computing (HPC) in cloud