In this post, we'll explore the benefits of transitioning from Anaconda to Miniconda and provide a step-by-step guide to help you make the switch.

The Data Science Setup

Traditionally, kickstarting your data science journey involved downloading and installing the comprehensive Anaconda distribution. This bundled package includes Python, over 200 packages, and additional tools like Anaconda Navigator and qtconsole.

A Leaner Alternative: Miniconda

While Anaconda simplifies the initial setup, its hefty size (around 600MB download, over 2GB installed) and inclusion of numerous unnecessary packages may not align with everyone's needs.

Enter Miniconda, a minimalist distribution that includes only Python, the essential conda tool, and a handful of core packages. With a download size of around 70MB (less than a tenth of Anaconda), Miniconda offers a more efficient and customizable data science environment. Any additional software needed can be easily installed later using conda.

Understanding Conda

There are a number of ways to set up your system to do data science in Python. In this tutorial, we will use a tool called conda. Conda is both a package manager and an environment manager.

In the context of Python, a package manager is a tool that installs, updates and removes third-party Python packages. A third-party Python package is any Python package that is not part of the Python standard library. The vast ecosystem of data science-related packages (e.g. pandas, NumPy, Matplotlib, scikit-learn, TensorFlow and more) make Python a go-to language for data enthusiasts.

Another popular package manager is pip, which is the default package manager for new Python installations. In this tutorial, we'll only touch on conda.

An environment manager creates an isolated container on your computer, known as a virtual environment, with their own installation of Python and third-party packages. This isolation ensures independence from any other Python installation on your machine.

Advantages of Migrating to Miniconda

1. Efficient Resource Utilization Anaconda comes preloaded with a lot of packages, some of which might be unnecessary for your specific workflows. Miniconda offers a minimalist approach, allowing you to install only what you need. This not only saves valuable storage space but also ensures your system resources are dedicated to the tools and libraries you actually tend to use. Okay, in Anaconda, you have the conda clean --all command, which basically cleans unused package cache and other files. However, Miniconda's smaller package footprint tends to be more efficient.

2. Simplified Environment Management The Anaconda Navigator, a graphical user interface became obsolete with the speed and simplicity of the command prompt or terminal. Creating a new environment might seem intimidating when you are just starting. However, you will quickly switch to the command prompt or terminal, since almost all tutorials you'll follow will use the command prompt or terminal. Embracing the command prompt or terminal will not only enhance efficiency but also integrate smoothly with your preferred code editor.

3. Streamlined Deployment Deploying your code on production servers can be a daunting task, especially when you are dealing with the unnecessary bloat from Anaconda. Miniconda simplifies the deployment process by allowing you to create virtual environments with only the essential packages, ensuring a clean and efficient runtime on production servers.

Step-by-Step Approach

1. (Optional) Export Environment Files

If you want to recover environments from Anaconda, export your environment files using the following commands:

conda activate your_env

Next, export your environment to a .yml file.

On Windows:

conda env export | findstr /V "^prefix: " > your_env.yml

On macOS or Linux:

conda env export | grep -v "^prefix: " > your_env.yml

Note on "prefix"

Note that conda creates a "prefix" in the .yml file. In the context of Conda environments, the "prefix" refers to the location on your file system where the Conda environment is installed. The "prefix" line in the your_env.yml file specifies the path to the directory where the environment is created.

For instance:

prefix: C:\Users\YourUsername\Anaconda3\envs\your_environment_name

This information is used when you or someone else later tries to recreate the Conda environment from the your_env.yml file. The presence of the "prefix" allows Conda to know where to create the environment.

When you export the environment using conda env export, you might want to exclude the "prefix" line from the exported YAML file if you plan to share the environment file across different systems or users. This is because the specific path may not be valid or appropriate on another system. That's why the grep -v "^prefix: " (or findstr /V "^prefix: ") is used—it removes the "prefix" line from the exported environment file.

2. Get Rid of Anaconda

Uninstall Anaconda by following the instructions here.

3. Install Miniconda

Find instructions on how to install Miniconda here.

4. Create Miniconda Environments

If you want a clean start, we recommend creating new environments. You can use the following command to create a new environment with the latest (at the time of writing) Python version:

cond create -n "your_env" python=3.12

Activate the environment with:

conda activate your_env

Alternatively, you can use the following command to recreate a predefined environment. Note that this will install all packages listed in the .yml file (potentially packages you don't need).

conda env create -f your_env.yml

If you want to specify a different install path than the default for your system (not related to 'prefix' in your_env.yml), just use the -p flag followed by the required path. Note that we previously deleted the "prefix" line. However, you can use this command if you want to install an environment file with the prefix line still in it.

conda env create -f your_env.yml -p /home/user/anaconda3/envs/env_name

5. Install Third-party Packages

Finally, we can install some packages in our new or recovered environment. I tend to install the pandas in almost al my environments. Let's install this packages in our activated environment.

conda install pandas

Congratulations! You've successfully made the transition from Anaconda to Miniconda. Stay tuned for more!

In this post, we'll take a practical look at importing various data types, such as File, Folder and Table types. We'll use the Azure Machine Learning Python SDK v2.

Prerequisites

• An Azure Subscription: If you don't have one, don't worry. You can create a free Azure account with a limited $200 USD credit to get started.
• An Azure Machine Learning Workspace: If you haven't set up your workspace yet, you can follow the quick start guide to create the necessary resources.
• The Azure Machine Learning Python SDK v2: To leverage the power of Azure Machine Learning, ensure that you have the Python SDK v2 installed.

Setting up Your Environment

To start importing data into your Azure Machine Learning workspace, you'll need to prepare your environment. We'll be using the mltable package for data loading, so make sure you install it in your Python environment. Run the following command to install the necessary packages:

pip install mltable azureml-dataprep[pandas]

Understanding Data Types and URIs

Azure Machine Learning allows you to work with three primary data types - Files, Folders, and Tables. Each data type serves a specific purpose, making it crucial to understand when and how to use them.

• File: read/write a single file, for instance storing a .pkl file of your serialized model on your local environment after training.
• Folder: read/write a folder of files, for instance importing a folder with images for training a deep-learning model.
• Table: read/write via a dynamic schema, for instance you want to start an AutoML job.

Each data type can be referenced by a Uniform Resource Identifier (URI). The URI represents the storage location on your local computer or on an external resource. There are different URIs for each storage location. Here are some URI examples:

• Azure Machine Learning Datastore: azureml://datastores/<data_store_name>/paths/<folder1>/<folder2>//<file>.csv
• Local computer: ./home/user/data/my_data
• Public http(s) server: https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv
• Blob storage: wasbs://<containername>@<accountname>.blob.core.windows.net/<folder>/
• Azure Data Lake Gen1: abfss://<file_system>@<account_name>.dfs.core.windows.net/<folder>/<file>.csv
• Azure Data Lake Gen2: adl://<accountname>.azuredatalakestore.net/<folder1>

Instead of using long storage paths of URIs, we'll create data assets, whereby we can access the data via a friendly name. The data asset is, just like the URI, a reference to the data source location, along with a copy of its metadata.

Creating Data Assets

Project structure:

project/
     |-- data/
     |-- mltable/
     |-- scripts/
     |-- config.json

Step 1: Setting up the MLClient

In this step, we'll instantiate the MLClient using a config.json file. Make sure you have the necessary configuration file in place, as shown below:

{
    "subscription_id": "<YOUR_SUBSCRIPTION_ID>",
    "resource_group": "<YOUR_RESOURCE_GROUP>",
    "workspace_name": "<YOUR_WORKSPACE_NAME>"
}

You can test your configuration by executing the following code in the Python interpreter in interactive mode.

>>> from azure.ai.ml import MLClient
>>> from azure.identity import DefaultAzureCredential
>>> ml_client = MLClient.from_config(DefaultAzureCredential())
>>> print(ml_client)

Step 2: Add Data

Add a file to the .\data directory. You can use the this titanic example dataset if you don't have any file.

Step 3: Create a File data asset

Navigate to the . \script folder and create a Python file named create_file_data_asset.py. This script will register the file as a data asset.

# create_file_data_asset.py
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential
from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

if __name__ == '__main__':

    ml_client = MLClient.from_config(DefaultAzureCredential())

    version = '<VERSION>'
    path = './data/<FILE_NAME>'
    name = '<DATA_ASSET_NAME>'
    description = '<DATA_ASSET_DESCRIPTION>'

    data_asset = Data(
        path=path,
        type=AssetTypes.URI_FILE,
        description=description,
        name=name,
        version=version
    )
    try:
        ml_client.data.create_or_update(data_asset)
        print(f'Dataset `{name}` version `{version}` created/updated successfully.')
    except Exception as e:
        print(f'Error: {str(e)}')

Step 4: Create a Table data asset

In the previous step, we registered the file as a File data asset. However, if your data is tabular, you can create an MLTable from it and register it as a data asset. An MLTable is a schema that defines a set of operations for loading data from the source. We'll author an MLTable file using mltable, which will document the data loading operations, and create an MLTable data asset.

# to_ml_table.py
import mltable
from mltable import MLTableHeaders, MLTableFileEncoding
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential


if __name__ == '__main__':

    ml_client = MLClient.from_config(DefaultAzureCredential())

    version = '<FILE DATA ASSET VERSION>'
    name = '<FILE DATA ASSET NAME>'

    data_asset = ml_client.data.get(name=name, version=version)

    tbl = mltable.from_delimited_files(
        paths=[{'file': data_asset.path}],
        delimiter=',',
        header=MLTableHeaders.all_files_same_headers,
        infer_column_types=True,
        include_path_column=False,
        encoding=MLTableFileEncoding.utf8
    )

    print(f'MLTable `{name}` version `{version}` created from delimted files successfully.')

    print(tbl.show(5))
    df = tbl.to_pandas_dataframe()
    print(df.head(5))
    tbl.save('./mltable/<MLTABLE NAME>')

You can take a look in the .\mltable\<MLTABLE NAME> folder to find a file called MLTable. This is a YAML file that describes the operations for data loading.

paths:
- file: azureml://subscriptions/<subscription_id>/resourcegroups/<resource_group_name>/workspaces/<workspace_name>/datastores/<datastore_name>/paths/LocalUpload/<path_id>/<file_name>
transformations:
- read_delimited:
    delimiter: ','
    empty_as_string: false
    encoding: utf8
    header: all_files_same_headers
    include_path_column: false
    infer_column_types: true
    partition_size: 20971520
    path_column: Path
    support_multi_line: false
type: mltable

You can load the MLTable as follows:

# load_ml_table.py
import mltable

if __name__ == '__main__':

    tbl = mltable.load('./mltable/<MLTABLE NAME>')
    print(tbl.show(5))

Next we can create a Table data asset using the following code:

# create_ml_table.py
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential
from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

if __name__ == '__main__':

    ml_client = MLClient.from_config(DefaultAzureCredential())

    path = './mltable/<MLTABLE NAME>'
    version = '<MLTABLE DATA ASSET VERSION>'
    name = '<MLTABLE DATA ASSET NAME>'
    description = '<MLTABLE DATA ASSET DESCRIPTION>'

    data_asset = Data(
        path=path,
        type=AssetTypes.MLTABLE,
        description=description,
        name=name,
        version=version
    )

    try:
        ml_client.data.create_or_update(data_asset)
        print(f'MLTable `{name}` version `{version}` created/updated successfully.')
    except Exception as e:
        print(f'Error: {str(e)}')

Step 5: Archive a data asset

Azure Machine Learning does not support data asset deletion due to its immutable nature. Instead, you can archive the data asset. Use the provided Python script to archive a previously created File Data Asset.

# archive_data_asset.py
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

if __name__ == '__main__':

    ml_client = MLClient(DefaultAzureCredential())

    name = '<DATA ASSET NAME>'

    ml_client.data.archive(name)

In this post, we've only scratched the surface of what Azure Machine Learning can do for your data management needs. There are many more exciting features and capabilities waiting to be explored. Stay tuned for more in-depth guides and tutorials on Azure Machine Learning, where we'll dive deeper into its functionalities.

In this post, we'll dissect why Azure Machine Learning should (or shouldn't) join your tech arsenal. This is just the start of a series of posts where we'll explore the pros and cons of Azure Machine Learning in greater depth. As we become more proficient in using this tool, the advantages and limitations will become even clearer. So, stay tuned for a deeper dive into the world of Azure Machine Learning! 🚀

Control over Model Training and Management

One of the standout features of Azure Machine Learning, in comparison with other Azure services related to data science (Azure AI Services and Azure Synapse Analytics), is the control it provides over model training and management. For seasoned data scientists, this is crucial. You get to customize and fine-tune every aspect of your machine learning workflows, including custom ML pipeline components, which are self-contained pieces of code that perform steps in the pipeline; custom training to train scikit-learn, TensorFlow, Keras, PyTorch, etc. models; and custom deployment to deploy a model to an endpoint in a custom container, etc.

But, and it's a reasonable "but" here 🤔, this level of control might be overwhelming. If you're just starting your journey in the world of data science, the sheer number of options, settings, and integrations in Azure Machine Learning could leave you scratching your head. In such cases, fear not! The Azure Machine Learning studio offers great functionality to help you kick start your machine learning adventures.

Ease of Getting Started

You can quickly implement a machine learning solution for well-known machine learning problems with AutoML. This can be completely done in the Azure Machine Learning studio, which allows you to manage your workload via a user interface. This is a great example of how to train a classification model with no-code AutoML using automated machine learning in Azure Machine Learning studio.

Comprehensive Lifecycle Management

Azure Machine Learning doesn't just stop at model training; it covers the complete machine learning lifecycle. The Azure Machine Learning studio is designed to be intuitive, making it easier for you to develop, train, deploy, and manage your models. This streamlined experience is advantageous for productivity and collaboration, especially in larger teams or organizations.

However, there's a flip side. Some data scientists who crave advanced customization or require specific tool integrations might find Azure Machine Learning slightly restrictive. It's a trade-off between ease of use and the ability to tailor every aspect of your workflow to your precise needs.

Version Management

Now, let's shift our focus to another critical aspect: version management. Version control can be a tricky issue in the world of machine learning. Keeping track of different model versions, datasets, and experiments is crucial but can be a logistical nightmare. Azure Machine Learning comes to the rescue here by automating model and data version management. This reduces the risk of conflicts and ensures that you can easily roll back to a previous state if something goes wrong.

In conclusion, Azure Machine Learning is a versatile tool for data scientists. It can be your trusty sidekick in the cloud, but like any tool, it's essential to weigh its benefits against your specific needs and preferences. In upcoming posts, I'll delve deeper into the core components of Azure Machine Learning, helping you determine if it's the right fit for your machine learning projects. Stay tuned!