MahendraPalla's

How to Set Up a Multi-Node Multi-Drive (MNMD) Minio Cluster - Production Ready.

mahendra's photo
mahendra
·

6 min read

The Enterprise Object Storage Feature Set

Hi All,

In this Article, we will set-up Minio 'Multi Node Multi Drive' (MNMD) Cluster.

What is Minio:

MinIO is a high-performance, S3 compatible object store. It is built forlarge scale AI/ML, data lake and database workloads. It is software-definedand runs on any cloud or on-premises infrastructure. MinIO is dual-licensedunder open source GNU AGPL v3 and a commercial enterprise license.

Minio can be used for various use cases such as:

  • Object storage for cloud-native and containerized applications.

  • Data backup and disaster recovery.

  • Big data and analytics.

  • Media and entertainment.

  • Archiving and long-term data retention.

Let's start setting up Minio Production Ready Cluster:

For this setup, I'm using 4 EC2 instances with 4 EBS volumes attached to each instance.

Below is the reference Architecture and the same can be used while deploying on On-Premises.

Why chosen the MNMD deployment.
MNMD deployments support erasure coding configurations which tolerate the loss of up to half the nodes or drives in the deployment while continuing to serve read operations. Use the MinIO Erasure Code Calculator when planning and designing your MinIO deployment to explore the effect of erasure code settings on your intended topology.

1) Server list.

NOTE: If you are deploying on AWS Ec2 Instance, select the private IPs here

Ec2-InstancePrivateIP-1 - 8 CPU and 16 Mem
Ec2-instancePrivateIP-2 - 8 CPU and 16 Mem
Ec2-InstancePrivateIP-3 - 8 CPU and 16 Mem
Ec2-InstancePrivateIP-4 - 8 CPU and 16 Mem

For all above 4 Ec2 machines, I have attached 4 EBS disk volumes.

2) Networking and Firewalls

Each node should have full bidirectional network access to every other node in the deployment. For containerized or orchestrated infrastructures, this may require specific configuration of networking and routing components such as ingress or load balancers. Certain operating systems may also require setting firewall rules. For example, the following command explicitly opens the default MinIO server API port 9000 for servers running firewallD :

NOTE: Make sure firewallD is running on the 4 nodes.

  • firewall-cmd --permanent --zone=public --add-port=9000/tcp

  • firewall-cmd --permanent --zone=public --add-port=9001/tcp

  • firewall-cmd --reload

3) Sequential Hostnames

MinIO requires using expansion notation {x...y} to denote a sequential series of MinIO hosts when creating a server pool. MinIO supports using either a sequential series of hostnames or IP addresses to represent each minio server process in the deployment.

This procedure assumes use of sequential hostnames due to the lower overhead of management, especially in larger distributed clusters.

Create the necessary DNS hostname mappings prior to starting this procedure. For example, the following hostnames would support a 4-node distributed deployment:

  • Open the /etc/hosts and update as below as we need to establish inter server connectivity.

4) Storage Requirement:

you can check the official documentation on the storage related.
https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.html#storage-requirements

5) Memory Requirements

MinIO recommends a minimum of 32GiB of memory per host. See Memory for more guidance on memory allocation in MinIO.

6) Time Synchronisation

Multi-node systems must maintain synchronized time and date to maintain stable internode operations and interactions. Make sure all nodes sync to the same time server regularly. Operating systems vary for methods used to synchronize time and date, such as with ntp, timedatectl, or timesyncd.

Check the documentation for your operating system for how to set up and maintain accurate and identical system clock times across nodes.

7) Erasure Coding Parity

Official documentation helps here.
https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.html#storage-requirements

8) Capacity-Based Planning

MinIO recommends planning storage capacity sufficient to store at least 2 years of data before reaching 70% usage. Performing server pool expansion more frequently or on a “just-in-time” basis generally indicates an architecture or planning issue.

For example, consider an application suite expected to produce at least 100 TiB of data per year and a 3 year target before expansion. By ensuring the deployment has ~500TiB of usable storage up front, the cluster can safely meet the 70% threshold with additional buffer for growth in data storage output per year.

Since MinIO erasure coding requires some storage for parity, the total raw storage must exceed the planned usable capacity. Consider using the MinIO Erasure Code Calculator for guidance in planning capacity around specific erasure code settings.

9) Recommended Operating Systems

This tutorial assumes all hosts running MinIO use a recommended Linux operating system such as RHEL8+ or Ubuntu 18.04+.

10) Install the MinIO Binary on Each Node

In this, Im using Amazon Linux machines. with below commands I able to install Minio on the 4 machines.

11) Once installed, Make sure the file 'minio.service' is automatically created in /usr/lib/systemd/system/minio.service, as shown in below in each node.

[Unit]

Description=MinIO

Documentation=https://min.io/docs/minio/linux/index.html Wants=network-online.target After=network-online.target AssertFileIsExecutable=/usr/local/bin/minio [Service] WorkingDirectory=/usr/local User=minio-user Group=minio-user ProtectProc=invisible EnvironmentFile=-/etc/default/minio ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi" ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

Restart=always

LimitNOFILE=65536

TasksMax=infinity

TimeoutStopSec=infinity

SendSIGKILL=no

[Install]

WantedBy=multi-user.target

12) Create Disks in all 4 Node:

  • Login to all Ec2 machine and run the command 'lsblk'. Make sure all Disks are attached to the mounts as shown below,

  • Create XFS disks

    • mkfs.xfs /dev/sdb

    • mkfs.xfs /dev/sdc

    • mkfs.xfs /dev/sdd

    • mkfs.xfs /dev/sde

  • Create a directory to mount to disks in all VMs.
    mkdir -p /mnt/disk1

    mkdir -p /mnt/disk2

    mkdir -p /mnt/disk3

    mkdir -p /mnt/disk4

  • Mount the directories to disks in all VMs.

    mount /dev/sdb /mnt/disk1
    mount /dev/sdc /mnt/disk2
    mount /dev/sdd /mnt/disk3

    mount /dev/sde /mnt/disk4

    and finally it should look like below in all VMs.

  • [root@minio-1]$ lsblk

    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS

    xvda 202:0 0 8G 0 disk

    ├─xvda1 202:1 0 8G 0 part /

    ├─xvda127 259:0 0 1M 0 part

    └─xvda128 259:1 0 10M 0 part /boot/efi

    xvdb 202:16 0 50G 0 disk /mnt/disk1

    xvdc 202:32 0 50G 0 disk /mnt/disk2

    xvdd 202:48 0 50G 0 disk /mnt/disk3

    xvde 202:64 0 50G 0 disk /mnt/disk4

  • create a folder 'minio' in all the disks.

    mkdir -p /mnt/disk1/minio

    mkdir -p /mnt/disk2/minio

    mkdir -p /mnt/disk3/minio

    mkdir -p /mnt/disk4/minio

13) The minio.service file runs as the minio-user User and Group by default. You can create the user and group using the groupadd and useradd commands. The following example creates the user, group, and sets permissions to access the folder paths intended for use by MinIO. These commands typically require root (sudo) permissions.

  • groupadd -r minio-user

  • useradd -M -r -g minio-user minio-user

  • chown -R minio-user:minio-user /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4 (Update the minio-user to all mounts in all VMs), it should look as below.

  • cd /usr/lib/systemd/system
    chown -R minio-user:minio-user minio.service

  • chown -R minio-user:minio-user /mnt/disk1/minio /mnt/disk2/minio /mnt/disk3/minio /mnt/disk4/minio

14) Create the Service Environment File

Create an environment file at /etc/default/minio. The MinIO service uses this file as the source of all environment variables used by MinIO and the minio.service file

create and edit the file./

vi /etc/default/minio and type the below content in it.

MINIO_VOLUMES="http://minio{1...4}.example.net:9000/mnt/disk{1...4}/minio"

# Use http here, as we didn't configured TLS to use HTTPS.

MINIO_OPTS="--console-address :9001"

MINIO_ROOT_USER=minioadmin

MINIO_ROOT_PASSWORD=minio-secret-key-CHANGE-ME

You can specify other environmental variables by using official link: https://min.io/docs/minio/linux/reference/minio-server/settings.html#minio-server-environment-variables

15) Run the MinIO Server Process

Minio should run with the below commands.

  • sudo systemctl enable minio.service

  • If Minio is not starting up , then you can check the Logs using below command to troubleshoot.
    journalctl -u minio.service

16) Open the MinIO Console

you can open Minio UI with any one of the Ec2 Instance:9001, you can configure Proxy infant of minio servers if needed (https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.html#networking-and-firewalls)

Ec2-public-ip:9001

17) you can connect to Minio using S3cmd or minio client or through UI.

for example, once you create a bucket in UI, the bucket will get placed in all 4 Nodes, 16 disks.

Play with the UI.

ENJOyyyyyyyyyyyyyyyyyyyy..........................

Thanks for Reading....