Most of the companies around the
globe using Netezza data warehouse appliance to perform their near real-time ,
complex analytics and data warehouse needs.
Read More Details : FPGA- FAST framework
Read More Details : FPGA- FAST framework
NPS performance is made of a
result of framework of FPGA Accelerated
streaming technology (FAST) engine. This engine is just like a turbo-charger to
an already powerful engine. This FAST engine what makes the Netezza a powerful
server. Netezza has introduced a FAST Engine, Compress, which uses technology
to deliver yet another 100-200% performance increase in streaming analytic
performance. Netezza’s Compress Engine is designed primarily for performance
improvement. Rather than the CPU-intensive compression efforts employed by
other vendors to reduce disk usage that also result in reduced performance, the
Compress Engine accelerates performance.
The FAST Engines Framework
The FPGA is a the heart of
Netezza’s patented streaming architecture. Netezza uses the FPGA to handle
important functionalities and filter out unwanted data during early stage of the
disk read. This process of early unnecessary data elimination improves the
performance as it frees up the downstream systems like I/O, CPU, memory and network. This enables NPS
to process only the required data and client request as soon as possible. Those
primary functions that are built into the FPGA in the NPS system are referred
to as "engines" and compose the FAST Engines framework.
The FPGA-Accelerated Streaming
Technology (FAST) Engines Framework is based on three main concepts:
· FAST Engines are
basic analytic functions electronically programmed into the FPGA to
accelerate query performance
· FAST Engines are
dynamically reconfigurable
· FAST Engines are
customized at run-time for each snippet executed in the SPU.
Dynamic re-configurability allows each of the
engines embedded in the FPGA to be modified, disabled or extended by the NPS
system at any point of time. Run-time customization enables the FPGA to
incorporate parameters passed to each engine to optimize the behavior of the
FPGA for a particular query snippet. By enabling this snippet-by-snippet
customization, one could think of the NPS platform as providing an ‘optimized
hardware configuration for each snippet’.
The five current FAST Engines in
the framework include Control, Parse,
Visibility, Project and Restrict.
These engines work in a combined serial and parallel manner.
Overall FAST framework is
depicted in below diagram.
Fig.1. Netezza FAST
engine framework
Control Engine
The Control Engine controls the
hard disk and manages direct memory access (DMA) data flows from the disk to
the SPU memory. It manages reads and writes to the disk drive and data flows to
memory. Storage management performance enhancement functions, such as Zone Maps
are enforced through the Control Engine.
Parse Engine
The Parse Engine performs two key roles in the FAST Engines framework.
First, it provides Error-Correcting Code (ECC)
checking and correction as data streams flows out of disk drive.
Second, it parses the incoming
data stream from the disk drive and passes the appropriate data to each of the
remaining three downstream engines such as Visibility, Project and Restrict.
Visibility Engine
The Visibility Engine provides the ACID
(Atomicity, Consistency, Isolation and Durability) isolation enforcement for
the NPS appliance. As data are streamed from the disk drive, the Visibility
Engine will removes rows of data that should not be visible to the particular
query snippet being executed - either because the records had been marked
deleted by an earlier query or because they had been added to the database
after the start of the current query. Data that should not be visible for
processing is immediately filtered away and eliminated in this FPGA engine.
Project Engine
The Project Engine enforces filtering of the
column data read from the disk based on the "SELECT" clause of the
SQL statement. Only columns included by the clause are allowed to continue for
further processing and any columns not selected will be filtered and eliminated
prior to any records being sent to the CPU and memory. This steps increases the
query performance as unwanted columns are already filtered out.
Restrict Engine
The Restrict Engine enforces filtering of
records that are read from the disk based on the "WHERE" clauses of
the SQL statement. Only records that satisfy the clauses will be allowed to
continue for further processing and any records not selected are filtered and
eliminated prior to being sent to the on-board CPU and memory
The Restrict
Engine typically results in an exceptionally high degree of data record
filtering, greatly reducing the follow-on snippet processing work required of
the CPU and memory on each SPU.
Compress Engine
The Compress Engine will boost streaming
query performance by 2-3x (100-200%) via software enhancement to the NPS
appliance. Compress Engine will be able to compress the data to use disk much
more effectively and increase query performance as data flows from the disk
drives. Netezza’s Compress Engine is designed primarily for performance
improvement, and not just to reduce data sizes.
The Compress Engine, like all
other FAST Engines, is dynamically configurable and optimized at run-time. As
data is written to disk (e.g., during data load, insert or update operations)
it is compressed into a compiled format, column-by-column with the original
data replaced by the Compress Engine instruction set for decompilation. As this
data is read from the disk, the Compress Engine reads its instruction set and
reassembles the original data as it streams from the disk, effectively raising
the streaming data rate by as much as 200% - this step increases effective
scanning rate per SPU node from over 60 MB/s to approximately 200 MB/s.
No comments:
Post a Comment