![DataWorks Summit](/img/default-banner.jpg)
- 230
- 327 254
DataWorks Summit
Приєднався 31 бер 2017
APACHE METRON INCUBATING AS A CASE STUDY OF A MODERN STREAMING ARCHITECTURE ON HADOOP
There have been many voices discussing how to architect streaming
applications on Hadoop. Before now, there have been very few worked
examples existing within the open source. Apache Metron (Incubating) is a
streaming advanced analytics cybersecurity application which utilizes
the components within the Hadoop stack as its platform.
Link to Slides: www.slideshare.net/Hadoop_Summit/bringing-it-all-together-apache-metron-incubating-as-a-case-study-of-a-modern-streaming-architecture-on-hadoop
We will attempt to go beyond theoretical discussions of Kappa vs Lambda
architectures and describe the nuts and bolts of a streaming
architecture that enables advanced analytics in Hadoop. We will discuss
the componentry that we had to build and what we could utilize. We will
discuss why we made the architectural decisions that we made and how
they fit together to knit together a coherent application on top of many
different Hadoop ecosystem projects.
We will also discuss the domain specific language that we created out of
necessity to enable a pluggable layer to enable user defined enrichments.
We will discuss how this helped make Metron less rigid and easier to
use. We will also candidly discuss mistakes that we made early on.
Speaker:
CASEY STELLA
Principal Software Engineer/Data Scientist
Hortonworks
Link to Slides: www.slideshare.net/Hadoop_Summit/bringing-it-all-together-apache-metron-incubating-as-a-case-study-of-a-modern-streaming-architecture-on-hadoop
Link to event session page: dataworkssummit.com/san-jose-2017/sessions/bringing-it-all-together-apache-metron-incubating-as-a-case-study-of-a-modern-streaming-architecture-on-hadoop/
applications on Hadoop. Before now, there have been very few worked
examples existing within the open source. Apache Metron (Incubating) is a
streaming advanced analytics cybersecurity application which utilizes
the components within the Hadoop stack as its platform.
Link to Slides: www.slideshare.net/Hadoop_Summit/bringing-it-all-together-apache-metron-incubating-as-a-case-study-of-a-modern-streaming-architecture-on-hadoop
We will attempt to go beyond theoretical discussions of Kappa vs Lambda
architectures and describe the nuts and bolts of a streaming
architecture that enables advanced analytics in Hadoop. We will discuss
the componentry that we had to build and what we could utilize. We will
discuss why we made the architectural decisions that we made and how
they fit together to knit together a coherent application on top of many
different Hadoop ecosystem projects.
We will also discuss the domain specific language that we created out of
necessity to enable a pluggable layer to enable user defined enrichments.
We will discuss how this helped make Metron less rigid and easier to
use. We will also candidly discuss mistakes that we made early on.
Speaker:
CASEY STELLA
Principal Software Engineer/Data Scientist
Hortonworks
Link to Slides: www.slideshare.net/Hadoop_Summit/bringing-it-all-together-apache-metron-incubating-as-a-case-study-of-a-modern-streaming-architecture-on-hadoop
Link to event session page: dataworkssummit.com/san-jose-2017/sessions/bringing-it-all-together-apache-metron-incubating-as-a-case-study-of-a-modern-streaming-architecture-on-hadoop/
Переглядів: 4 820
Відео
Enabling Modern Application Architecture using Data gov open government data
Переглядів 7067 років тому
Enabling Modern Application Architecture using Data gov open government data
Introduction to Deep Learning and AI at Scale for Managers
Переглядів 5737 років тому
Introduction to Deep Learning and AI at Scale for Managers
Analyst Panel Unravels the Data Industry
Переглядів 547 років тому
Analyst Panel Unravels the Data Industry
Securing Enterprise Healthcare Big Data by the Combination of Knox F5, Ranger, TFA and Kerberos Coup
Переглядів 1927 років тому
Securing Enterprise Healthcare Big Data by the Combination of Knox F5, Ranger, TFA and Kerberos Coup
Schema Registry - Set your data free
Переглядів 11 тис.7 років тому
Schema Registry - Set your data free
Security, ETL, BI & Analytics, and Software Integration
Переглядів 3657 років тому
Liberty Mutual Enterprise Data Lake Use Case Study Link to slides: www.slideshare.net/Hadoop_Summit/security-etl-bi-analytics-and-software-integration By building a data lake, Liberty Mutual Insurance Group Enterprise Analytics department has created a platform to implement various big data analytic projects. We will share our journey and how we leveraged Hortonworks Hadoop distribution and oth...
HDFS Tiered Storage Mounting Object Stores in HDFS
Переглядів 5347 років тому
HDFS Tiered Storage Mounting Object Stores in HDFS
APACHE HADOOP YARN PRESENT AND FUTURE
Переглядів 7197 років тому
This session is a (Advanced) talk in our Apache Hadoop track. It focuses on Apache Hadoop, Docker / Container and is geared towards Architect, Developer / Engineer, Operations / IT audiences.
Continuous Data Ingestion pipeline for the Enterprise
Переглядів 2,3 тис.7 років тому
Continuous Data ingestion platform built on NIFI and Spark that integrates variety of data sources including real-time events, data from external sources , structured and unstructured data with in-flight governance providing a real-time pipeline moving data from source to consumption in minutes. The next-gen data pipeline has helped eliminate the legacy batch latency and improve data quality an...
WHOOPS, THE NUMBERS ARE WRONG! SCALING DATA QUALITY @ NETFLIX
Переглядів 6 тис.7 років тому
Netflix is a famously data-driven company. Data is used to make informed decisions on everything from content acquisition to content delivery, and everything in-between. As with any data-driven company, it’s critical that data used by the business is accurate. Or, at worst, that the business has visibility into potential quality issues as soon as they arise. But even in the most mature data war...
WHAT’S POSSIBLE WITH AI AND DATA IN 2017?
Переглядів 2217 років тому
This session is a talk in our Enterprise Adoption track. It focuses on Apache Spark, Other and is geared towards CXO, Architect, Data Scientist, Developer / Engineer audiences.
Semi Supervised Learning In An Adversarial Environment
Переглядів 1,1 тис.7 років тому
Semi Supervised Learning In An Adversarial Environment
Large Scale Graph Processing & Machine Learning Algorithms for Payment Fraud PrevenUon
Переглядів 2417 років тому
Large Scale Graph Processing & Machine Learning Algorithms for Payment Fraud PrevenUon
Governance Bots Metadata Driven Compliance Through AI, Atlas and NiFi
Переглядів 8577 років тому
Governance Bots Metadata Driven Compliance Through AI, Atlas and NiFi
DRUID SUB SECOND OLAP QUERIES OVER PETABYTES OF STREAMING DATA
Переглядів 3,9 тис.7 років тому
DRUID SUB SECOND OLAP QUERIES OVER PETABYTES OF STREAMING DATA
How Big Data and Predictive Analytics are revolutionizing AML and Financial Crime Detection
Переглядів 9 тис.7 років тому
How Big Data and Predictive Analytics are revolutionizing AML and Financial Crime Detection
HANDLING KERNEL UPGRADES AT SCALE - THE DIRTY COW STORY
Переглядів 4027 років тому
HANDLING KERNEL UPGRADES AT SCALE - THE DIRTY COW STORY
Beyond unit tests Deployment and testing for Hadoop Spark workflows
Переглядів 5537 років тому
Beyond unit tests Deployment and testing for Hadoop Spark workflows
End to end Data Governance with Apache Avro and Atlas
Переглядів 13 тис.7 років тому
End to end Data Governance with Apache Avro and Atlas
AN APACHE HIVE BASED DATA WAREHOUSE
Переглядів 3,5 тис.7 років тому
AN APACHE HIVE BASED DATA WAREHOUSE
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 mins without writing any
Переглядів 2,1 тис.7 років тому
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 mins without writing any
Future Architecture of Streaming Analytics Capitalizing on the Analytics of Things AoT
Переглядів 1897 років тому
Future Architecture of Streaming Analytics Capitalizing on the Analytics of Things AoT
The Unbearable Lightness of Ephemeral Processing
Переглядів 507 років тому
The Unbearable Lightness of Ephemeral Processing
Hive is dead
Thanks🎉
Write Audit Publish pattern at 16:34 , good talk
20:40 Why would proxyClient write to Kafka cluster when you have Kafka Rest proxy for that job? Also as per your explanation both Local Agent and ProxyClient are doing the same job. batching and sending the messages when downstream recovers.. you should have explained with an example..
You guys know soo much wy don't u create some tutorials in UA-cam or udemy to contribute to other associates
@2:05 I thought I was watching Dr Katz
it is in which langauage #java or #python
It's in Java.
Sir, Is online learning algorithm same as self learning algorithm ?
Is there masked dataset can be tested for proving false positives in SAR or STR
Enjoyed this talk so much!
How about saving state for Session windows? Currently, looks like Beam doesn't support Stateful processing for Merged windows, any other approach to achieve same?
Amazing! 👏👏👏
🔥🔥🔥🔥
I use Impala query (on HUE) in order to transform and load data into Kudu tables but how can I execute these queries autoamatically in order to update a kudu table on daily basis? I would very much appreciate your help, many thanks
Great thanks
really concern about the comparison on query performance data among Spark SQL, Hive, Presto etc. How can presto be that slower than Hive???
Wow, thats physically painful to watch.
Note for (some) developers: This is NOT a video about Yarn - the software package manager for JavaScript/TypeScript. This is about a computing cluster technology called YARN.
wrr
These talks are so useless, they don't discuss any problems or whatever lead to the use of these techs than the other. anyone can come up with this HLD
Thank you for sharing this video.
Really enjoyed this talk !
Simple and crisp explanation.Many Thanks
Doesn't talk about SSL scaling, caching and how the application picks up the DC. The author keeps saying aye as though it's a pirate movie 😂, like it.
very well explained...
Thank You for this video.
Oh, I wish there were a transcript of this! Who has time to watch a video?
Extremely confusing.
Don't recommend to use NIFi for data ingestion. It's difficult to work with flow provisioning / IT testing is almost impossible, using existing components and building flows become a nightmare when it grows. No easy way to add ingestion as a pipeline into the existing system (like Oozie). Flow changes are difficult to track (no integration with git out of the box).
i completed the html certification from refsnes data. w3shools. paid for the certificate. did not receive my certificate. where do i go from here.
book reading
Hi Moon, Thanks for the wonderful video !!! Is it mandatory to keep the packages inside zeppelin/helium or we can keep it any where. Please suggest which property need to set in zeppelin-site property file. I have a requirement to add TOC(zeppelin-toc-spell) package and it a part of Zeppelin-Spell.
This video clearly mention what is the architecutre of uber. But did not said how it evolve to trillion event
Can we estimate from this how much gps data a particular road is recieving on real time? Need your prophetic remarks thank you
Impressive architecture...
Yeah, real innovation here. Use the fuck out of kafka. Well done
great talk Jeremy!
Thank you, Bhai!!
Sounds like the mic wasn't quite happy there
Very helpful, thank you
Hi Muhammad Imran! How would you charge me for private online lessons or actually solve some problems with SparkR? I could pay you by Paypal or even by hiring you as a freelancer
www.slideshare.net/HadoopSummit/file-format-benchmark-avro-json-orc-parquet
Detailed but concise presentation, a great combination and indeed extremely helpful. Thank you for sharing.
do you consider drivers are machine? from these kind of techs humans are not humans anymore....
Good informative session. If you are interested only in Presto (Not Teradata) listen till 19th minute.
The introduction is way too long and tangential. Please make your presentations more concise in the future
i think you were probably not listening ... presenter did mention that there are a lot of ppl in the audience who dont have much background.
why does the video flicker so badly? :(
I think it was recorded from a projector's display.
Sound issues. But the volume of messages makes sound quality irrelevant.
Too Good one
Had not noticed the sound until halfway after which I read the comments!!
Haha its hard to unnotice the sound