Thet Paing

0 followers · 14 following · 321 views

on the atlas · 36

highlights · 429

If the unique key already exists in the destination table, the merge will update the record. And if the records don’t exist, merge will insert them.
dbt Incremental: Choosing the Right Strategy — P1 | by Alice Bui | Joon Solutions | Medium
Working closely with Backend Engineers on data logging. Bridging the disconnect between raw data and user needs. Collaborating with Machine Learning Engineers. Building innovative pipelines to refine models.
Why I Love My Job as a Data Engineer - YongJin Lee
As a midfielder, I was the link between offense and defense, influencing the game’s flow. Similarly, as a data engineer, I bridge the gap between technical and non-technical teams, playing a pivotal role in diverse projects.
Why I Love My Job as a Data Engineer - YongJin Lee
A common reason to use dependency functions over bit-shift operators is to create dependencies for tasks that were created in a loop and are stored in a list.
Manage task and task group dependencies in Airflow | Astronomer Documentation
A common reason to use dependency functions over bit-shift operators is to create dependencies for tasks that were created in a loop and are stored in a list.
Manage task and task group dependencies in Airflow | Astronomer Documentation
Upstream task: A task that must reach a specified state before a dependent task can run. Downstream task: A dependent task that cannot run until an upstream task reaches a specified state.
Manage task and task group dependencies in Airflow | Astronomer Documentation
It optimizes space as fact tables should not include low-cardinality or text fields. It mainly includes measures, foreign keys, and degenerate dimension keys.
Junk Dimension | Gability
It used to reduce the number of dimensions (low-cardinality columns) in the dimensional model and reduce the number of columns in the fact table. It is a collection of random transnational codes, flags, or text attributes.
Junk Dimension | Gability
Sensors: It’s a special subclass of Operators that are responsible for waiting for an external event to happen. TaskFlow-decorated @task: It is a custom Python function packaged up as a Task.
Airflow Architecture: Key Components & Best Practices
Operators: These are the predefined tasks that you can use to build most parts of your DAGs.
Airflow Architecture: Key Components & Best Practices
If you provide the --full-refresh flag to dbt run, dbt will treat incremental models as table models.
About dbt run command | dbt Developer Hub
To push a value to XCom return it at the end of your task as with traditional operators. To retrieve a value from XCom provide the object created by the upstream task as an input to your downstream task.
Pass data between tasks | Astronomer Documentation
XCom is a built-in Airflow feature. XComs allow tasks to exchange task metadata or small amounts of data. They are defined by a key, value, and timestamp.
Pass data between tasks | Astronomer Documentation
Sequential executor: The Sequential executor runs a single task instance at a time in a linear fashion with no parallelism functionality (A → B → C)
Airflow Executors | Astronomer Documentation
If you're running native Airflow, adopting a Celery executor means you'll have to set up an underlying database to support it (RabbitMQ/Redis)
Airflow Executors | Astronomer Documentation
At its core, the Celery executor is built for horizontal scaling.
Airflow Executors | Astronomer Documentation
The Local executor is ideal for testing.
Airflow Executors | Astronomer Documentation
Running Apache Airflow on a Local executor exemplifies single-node architecture.
Airflow Executors | Astronomer Documentation
The CeleryExecutor, for example, will by default run a max of 16 tasks concurrently.
Airflow Executors | Astronomer Documentation
The executor works closely with the scheduler to determine what resources will actually complete those tasks (using a worker process or otherwise) as they're queued.
Airflow Executors | Astronomer Documentation
The scheduler reads from the metadata database to check on the status of each task and decide what needs to get done and when.
Airflow Executors | Astronomer Documentation
The Metadata Database keeps a record of all tasks within a DAG and their corresponding status (queued, scheduled, running, success, failed, and so on) behind the scenes.
Airflow Executors | Astronomer Documentation
Document every resource, endpoint, request type, expected input, and response format. Try using GenAI-powered tools like Swagger to automate and standardize the documentation.
REST API: Everything You Need to Know
Use authentication (e.g., OAuth, API keys) to control access and HTTPS to encrypt data in transit.
REST API: Everything You Need to Know
For each resource, define the endpoints (URLs) and the HTTP methods (GET, POST, PUT, DELETE) that will be supported, aligning with the principles of REST for resource manipulation.
REST API: Everything You Need to Know
Start by identifying the resources (data entities) the API will expose, such as users, products, or services.
REST API: Everything You Need to Know
A better approach is to use a boolean flag to store the detection of an event and then handle it in the main loop.
Interrupts RPi Pico MicroPython : React to events
When an interrupt is enabled, interruption_handler() is called with the pin where the event occurred as the input argument.
Interrupts RPi Pico MicroPython : React to events
The code uses the Pin.irq() function, which takes a signal from a pin_button and converts it into an interrupt request.
Interrupts RPi Pico MicroPython : React to events
Interrupts provide a quick and easy way of responding to events without constantly checking a pin’s value. Whenever pin changes are detected, a defined function is automatically performed.
Interrupts RPi Pico MicroPython : React to events
the detection hardware is more responsive than the MicroPython script
Interrupts RPi Pico MicroPython : React to events
Doing anything else in the loop can cause the second problem, where a button press may go unnoticed. This is known as a “missing event.”
Interrupts RPi Pico MicroPython : React to events
The first problem is that the script must constantly check the value of the pin_button pin to determine if the button has been pressed.
Interrupts RPi Pico MicroPython : React to events
Fixed-Size: Because other data may follow immediately after the block of memory occupied by your array, expanding the block would overwrite that data, potentially causing severe errors or malfunctions.
Python's Array: Working With Numeric Data Efficiently – Real Python
Contiguous: Array elements are adjacent to each other in the computer memory, occupying a contiguous block of memory space
Python's Array: Working With Numeric Data Efficiently – Real Python
Homogeneous: All elements in an array share a common data type, allowing them to have a uniform size.
Python's Array: Working With Numeric Data Efficiently – Real Python
array data structure special is a few unique features
Python's Array: Working With Numeric Data Efficiently – Real Python
The list abstract data type is a linear collection of values forming an ordered sequence of elements.
Python's Array: Working With Numeric Data Efficiently – Real Python
That’s because the array is a specific data structure representing the list abstract data type.
Python's Array: Working With Numeric Data Efficiently – Real Python
abstract data types define the desired semantics, data structures implement them, and data types represent data structures in programming languages as built-in syntactic constructs.
Python's Array: Working With Numeric Data Efficiently – Real Python
The word abstract in this context means these data types leave the implementation details up to you, only defining the expected semantics or the set of available operations that an ADT must support. As a result, you can often represent one abstract data type using a few alternative data structures, which are concrete implementations of the same conceptual approach to organizing data.
Python's Array: Working With Numeric Data Efficiently – Real Python
The word abstract in this context means these data types leave the implementation details up to you, only defining the expected semantics or the set of available operations that an ADT must support. As a result, you can often represent one abstract data type using a few alternative data structures, which are concrete implementations of the same conceptual approach to organizing data.
Python's Array: Working With Numeric Data Efficiently – Real Python
official documentation
The Python pickle Module: How to Persist Objects in Python – Real Python
the s at the end of the function name stands for string.
The Python pickle Module: How to Persist Objects in Python – Real Python
However, it’s also faster and it works with many more Python types right out of the box, including your custom-defined objects.
The Python pickle Module: How to Persist Objects in Python – Real Python
It differs from the json module in that it serializes objects in a binary format, which means the result is not human readable.
The Python pickle Module: How to Persist Objects in Python – Real Python
Face Encoding: After alignment, facial features need to be transformed into a numerical representation that can be used for recognition. This process is called face encoding or face embedding.
Face Recognition in Python: A Comprehensive Guide | by Basil CM | Medium
Face Alignment: Once the faces are detected, it is essential to align them to a standardized pose. Face alignment techniques aim to normalize the face’s orientation, scale, and pose to improve consistency.
Face Recognition in Python: A Comprehensive Guide | by Basil CM | Medium
Face Detection: The first step is to detect faces within an image or video frame. This can be done using algorithms like Haar cascades, HOG (Histogram of Oriented Gradients), or deep learning-based models such as SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once).
Face Recognition in Python: A Comprehensive Guide | by Basil CM | Medium
Data modeling is done using SQLMesh and orchestrated in dagster. Complex transformations that require distributed work with large-scale joins run in Snowflake. One example is joining all the events from the first step in order to create a wide table where each row represents everything we know about a single auction.
Building an open data pipeline in 2024 - by Dan Goldin