Docker-mining on dedicated servers.

Face/Off · 2021-09-10 15:38:51

Hi all, I’m Face/Off, in this tutorial I will show you step by step how to run UAM in a docker-container on linux.
A small remark: I'm not very handy with creating documents and I've never been able to insert pictures properly, so only text so far.

First we have to decide on a Linux distribution. There are many of them, but if you use your computer solely as a node for the Utopia network, I recommend installing firmware-debian-10.11-amd64-netinst (this version contains additional proprietary drivers for your hardware), a distribution featuring minimal installation of additional software, which means we will give maximum computer resources to uam-bots.

Installing and configuring a Linux system for the Utopia network node (Part 1)

So, here we go:

Download the distribution package: https://cdimage.debian.org/cdimage/arch … etinst.iso

Save it to a convenient medium (usb stick, cd-disk). You can google how to do this.

Set it up. Out of old habit, I choose the usual NOT graphical installer. Select the Install option and let's go:

● Choose your language and keyboard layout (I am more used to English and US layout).
● Next, DHCP will automatically configure your IP address and subnet mask, but if you want to avoid it, click <Cancel> and configure manually.
● Enter a host name (whatever you like) and press <Continue>.
● Domain name can be skipped, since we are not going to deploy a web site or mail server on this computer - leave the output field blank. Click <Continue>.
● Root password, this is an important thing, come up with a complex but memorable password. I highly recommend NOT writing it down on a piece of paper though...
● Next is your username, but we are on an anonymous network, so use your usual nickname and click <Continue>.
● Repeat to create a new user (just press Enter).
● Enter the password for the new user.
● In the next dialog box, repeat the password entry, passwords must of course be the same.
● Next, choose a time zone, in principle any time zone will do for our purposes, so you can simply press Enter.
● In the next window, you need to partition your hard drive (IMPORTANT: it should be empty, preferably new and at least SSD or SSD NVME) into partitions, to avoid confusion just select Guided - use entire disk and press Enter four times.
● When the installer asks you: Write the changes to disk? - Select Yes and press Enter. This completes the disk installation process and the installer will install the base system.
● Later on the installer will ask you: Scan another CD or DVD? But since we have only one CD and it's a minimal one, just click No.
● Next, the installer will ask you to select a mirror from which it will later install all the necessary programs, as well as proxy settings (if you have one on your network, you'd better not ), just hit Enter three times.
● After selecting a mirror and proxy settings, the installer will ask you if you want to participate in the system improvement program (do we need it?), hit No.

And finally we finally got to the end:
● Select the required software suites for further installation, removing all asterisks except SSH server and standard system utilities. Click <Continue>
The installer will download a bunch of required software from the Internet (we have a NetInst-distribution)
and ask Install the GRUB boot loader to the master boot record? Click Yes and then select the disk you just installed Debian Linux on from the list. For example, for me it's "/dev/sda" and hit Enter again.
● We wait for the system to install the bootloader and offer to restart the system, click <Continue> and reboot.

This completes the basic installation of Linux on your bot farm. It gets much more interesting from here... and more confusing, but we'll try to wade through the thorns to the stars.

Installing and configuring a Linux system for the Utopia network node (Part 2)

● login root / your root password (hope you haven't forgotten it?).

Next you need to update the system to the latest versions and install the required software, this is done with a single command:

apt update && apt full-upgrade -y && apt install ethtool miniupnpd nload mc screen htop docker.io -y

The miniupnpd package will ask if it is started automatically at boot. Choose Yes, it will also ask for the name of your external and internal network, the former will be typed in immediately, the latter will be docker0 and hit Enter. All other packages will be installed silently and without any questions.

That's it, your system is fully up to date and all the necessary software is installed. Yes, that's the magic of Linux.
Now we need to configure our software and start "doing things".

Run Midnight Commander, it is a console file manager, it helps to see more clearly where you are, a lot like Far Manager or Norton Commander (for oldtimers).
Command: mc
To start, we configure the miniupnpd and docker packages.

● Go to /etc/systemd/system/multi-user.target.wants/
There we are interested in the @miniupnpd.service file, open it with the F4 button and correct one line:
Was: After=network-online.target
Became: After=network-online.target docker.service
Save (F2, same as in Far manager), exit.

● Reread service file:

systemctl daemon-reload

This will not only have miniupnpd loaded after all network drivers are loaded and the network
drivers themselves activated, it also adds the criterion of loading after docker is loaded, which
initializes docker0 virtual network driver. And after this miniupnpd is loaded and although it is
loaded, but not finding the network interface it immediately crashed with an error.

● Go to /etc/miniupnpd
See miniupnpd.conf file
Find string: #secure_mode=yes
Uncoment this line by removing the # sign in front of secure_mode=yes
Save (F2), exit.

This setting will force miniupnpd to only receive requests from the internal docker container network and prevent it from being used as a DDoS booster.

● Next we go to /etc/miniupnpd/
This is where miniupnpd_functions.sh
Fix firewall binary naming bug (yes, F4):
Was: IPTABLES=$(which iptables)
Became: IPTABLES=$(which iptables-legacy)
Just in case, let's also fix this:
Was: IPTABLES=$(which ip6tables)
Became: IPTABLES=$(which ip6tables-legacy)
Save (F2), exit.

● Another little detail I almost forgot about:
In /etc/sysctl.conf you need to uncomment the line #net.ipv4.ip_forward=1
Open sysctl.conf in file manager (F4) and remove # at the beginning of the line, save the file (F2).

This manipulation will enable gateway mode for our future docker containers.
We have now fixed the miniupnpd package a bit and it should now run and work properly with the docker.
And I'm too lazy to explain how to restart the services there, you can just reboot, type in the console: reboot and hit Enter. And then reboot, actually.

Loading...|

● Login as root and go ahead.
● Now we need to build an image for the docker containers.
Create an empty file:

touch Dockerfile

Type in this:

FROM debian:buster-slim
RUN cd /tmp && \
apt update && \
apt full-upgrade -y && \
apt install wget libglib2.0-0 netbase -y && \
wget https://update.u.is/downloads/uam/linux/uam-latest_amd64.deb && \
dpkg -i /tmp/uam-latest_amd64.deb
CMD /opt/uam/uam --pk %YOU_UTOPIA_KEY% --no-ui

where %YOU_UTOPIA_KEY% is your utopia public key.
and create an image:

docker build -t uam:latest .

(DOT AT THE END AND A SPACE BETWEEN "t" AND THE DOT IS REQUIRED)

● That's it, the image is created, it will weigh about 200 megabytes, which isn't much, now we need to make a couple of containers with manners:

docker run -d --restart always --cap-add=IPC_LOCK uam:latest

That's your first docker container is running, you can see the result in 15-30 minutes in utopia client on the Mining tab.
This last command can be repeated as many times as you need, the main thing to keep in mind is that your computer is not rubbery, and bots eat up resources quite decently.

Hints:

The number of containers can be seen with the command: docker ps
You can also check the load with the utilities: htop and load -m
Make sure that the Load Average in htop does not exceed 10-20, otherwise bots will drop
out, which is bad for the network. Ideally it should be less than a hundred at all, so don't be greedy.

Shaping

If you were still greedy and you had a high load average (you ran too many containers with a mining bot), then it makes sense to limit the network bandwidth of each bot, for this you need to take the following steps:
● Install wondershaper:

apt install wondershaper -y

● Create a script file:

touch shaper.sh

● We add these lines to it:

#!/bin/bash

for i in `ip a | grep veth | awk '{print $2}' | tr ':' ' ' | tr '@' ' ' | awk '{print $1}'`
do
echo "wondershaper $i 2048 2048".
wondershaper $i 2048 2048
done

● Change rights and run:

chmod 700 ./shaper.sh && /bin/bash ./shaper.sh

Do not forget that all manipulations must be done with root superuser rights.
Thus, each virtual network interface of the docker container will be limited to 2 Mbps for input and output.

Thanks for this simple and elegant solution:
sfdpmf: D9D2FFD70D50519707C3C7DCBE7D6F52DCFB3746432B7E78E7BE0D972B139E6B

Hetzner abuse reports

If you rent servers from Hetzner, then one day you will receive an abuse report.
If you want to fix this problem, you should do this:

● Install the iptables-persistent package: apt install iptables-persistent -y and answer <Yes> when the installer asks you to keep the existing rules.
● Go to /etc/iptables/ and find a file named rules.v4 and edit it.

Find string:

-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2

and insert BEFORE the line this commands:

-A DOCKER-ISOLATION-STAGE-1 -d 10.0.0.0/8 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-ISOLATION-STAGE-1 -d 100.64.0.0/10 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-ISOLATION-STAGE-1 -d 172.16.0.0/16 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-ISOLATION-STAGE-1 -d 192.168.0.0/16 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable

you should see this:

-A DOCKER-ISOLATION-STAGE-1 -d 10.0.0.0/8 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-ISOLATION-STAGE-1 -d 100.64.0.0/10 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-ISOLATION-STAGE-1 -d 172.16.0.0/16 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-ISOLATION-STAGE-1 -d 192.168.0.0/16 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2

Also notice the line:

:FORWARD DROP [0:0]

If it says DROP, then replace it with ACCEPT in order to allow packet forwarding.

● Then you must upload this to iptables:

service netfilter-persistent reload

So now packets coming from docker containers to private addresses described in RFC 1918 should be blocked.

You can check this:

iptables -vnL

If you see this, then you have achieved success:

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- docker0 !docker0 0.0.0.0/0 10.0.0.0/8 reject-with icmp-port-unreachable
0 0 REJECT all -- docker0 !docker0 0.0.0.0/0 100.64.0.0/10 reject-with icmp-port-unreachable
0 0 REJECT all -- docker0 !docker0 0.0.0.0/0 172.16.0.0/16 reject-with icmp-port-unreachable
0 0 REJECT all -- docker0 !docker0 0.0.0.0/0 192.168.0.0/16 reject-with icmp-port-unreachable

To automatically load these rules on server reboot, you need to fix one line in /etc/systemd/system/multi-user.target.wants:
We need the @netfilter-persistent.service file and change there:
Was: After = systemd-modules-load.service local-fs.target
Became: After = systemd-modules-load.service local-fs.target docker.service

Reread this file:

systemctl daemon-reload

Remark: iptables-persistent package allows you not to think about how to save the rules when you restart the server.
After loading the rules, the server does not need to be rebooted, the rules are added immediately.

Send donations and thanks to:
Cryptocard: FACE0002FC5FE972

Yes, I remember this, but there is no time yet and there are more urgent things:
"Later in the tutorial I'll add a script to automatically deploy bots, and also tell you how to attach
the docker container management webmode - portainer.io"

Thanks for help in translating this guide into English:
Makedonskiy - 4FB62131A403EE7D00C0ECAA85D68A6F8C21B717023B45EF8B26F81C03DF1A18
MasterOfCat - A5169F827554A47437E6E131E5FCA2EB9FA89127AC44E68FA21B51CC1FDF0626

UPD: updated docker run command, rewrote private address filter

Last edited by Face/Off (2022-02-01 17:59:49)

Saahil · 2021-09-10 16:36:39

Great work. This is very well detailed mining guide

Makedonskiy · 2021-09-20 08:39:09

indeed, a lot of work has been done. Docker is an ASIC for utopia, it is difficult, but the output performance exceeds all expectations

youtube · 2021-09-20 08:56:36

This is a unique guide for the average user, since not everyone could understand how to configure docker, I hope someday someone will also release a guide on how to configure docker on Windows

Unstressed · 2021-09-20 16:40:45

You're cool, FACE//OFF. Keep up the good work! I have uploaded a lot of useful information here.

Makedonskiy · 2021-09-23 14:30:12

it remains to create a list of data centers where they do not ask questions about the huge traffic when using the docker

iTruji · 2021-11-04 18:32:25

Does this guide works after the new requeriments of RAM?
I'm noob at the linux os.

Face/Off · 2021-12-23 12:25:17

iTruji;1003 wrote:

Does this guide works after the new requeriments of RAM?
I'm noob at the linux os.

I proceed from the standard system requirements for 1 bot: 4 cores, 4 gigabytes of RAM. In order for the OS to work and the bot to work stably, I recommend increasing the amount of RAM to 6 gigabytes.

money_ranger · 2022-01-04 17:19:53

I'm sorry to bother you, but i tried following your guide and i can't get incoming connections to the docker.

I' am through a VM in Virtual Box (where a single instance of Ubuntu is able to mine correctly) and i tried your distro and a ubuntu one, and both don't work. If i start a docker with the --network=host the miner start receiving incoming connections so i guess the problem is with the miniupnpd.

The UAM says it detected Upnp and the node is switched on, but even after 1 hr i have no incoming connections.

EDIT: i was trying to see if i could make it work on the VM as a test. Eventually im in the middle of buying some hardware and make some dedicated servers for utopia. So this may be a non issue. But i would still gladly appreciate some feedback on the situation.

Last edited by money_ranger (2022-01-04 20:02:01)

Dr-Hack · 2022-01-05 18:37:35

This is such a well written and explained guide !!! i am sure many will find it helpful

Dr-Hack · 2022-01-05 18:39:38

money_ranger;1297 wrote:

I'm sorry to bother you, but i tried following your guide and i can't get incoming connections to the docker.

If you have waited for about a day and still no incoming connection in that case you may wanna reconsider changing your VPS / VM as issue can be with that too ..

Face/Off · 2022-01-08 14:46:53

money_ranger;1297 wrote:

I'm sorry to bother you, but i tried following your guide and i can't get incoming connections to the docker. I' am through a VM in Virtual Box (where a single instance of Ubuntu is able to mine correctly) and i tried your distro and a ubuntu one, and both don't work. If i start a docker with the --network=host the miner start receiving incoming connections so i guess the problem is with the miniupnpd. The UAM says it detected Upnp and the node is switched on, but even after 1 hr i have no incoming connections.EDIT: i was trying to see if i could make it work on the VM as a test. Eventually im in the middle of buying some hardware and make some dedicated servers for utopia. So this may be a non issue. But i would still gladly appreciate some feedback on the situation.

This guide is written for dedicated or bare metal servers.
You are using a virtual machine
, read what a double nat is, and the general theory of network administration in linux

I can briefly explain, in your case you have two layers of networks through which you need to send network packets:
The first is the docker network, and it is written about this in the manual, i.e. you have overcome one nat.
The second is the network in which your computer is located, if it does not have a white IP and is located behind the router, that you need to solve this problem yourself

money_ranger · 2022-01-09 15:07:52

i don't understand: if 1 VM with 1 UAM works with a bridged connection through the VM why wouldnt the dockers works through the same setup? If your guide ''solves'' the NAT problem for the dockers?

You say the second network has a NAT too, but why isn't that an issue if i use just 1 Instance of the miner without the dockers?

Is it because the UAM resolves the NAT mapping ports through the Upnp only once? So basically your guide adds a virtual Upnp in the linux enviroment and the UAM maps through it natively just as it maps through my router NAT with his Upnp?

So basically if the UAM goes thorugh the Linux NAT enviroment it can't get through my Router NAT enviroment again?

Last edited by money_ranger (2022-01-09 15:16:04)

Face/Off · 2022-01-09 23:31:01

money_ranger;1388 wrote:

i don't understand: if 1 VM with 1 UAM works with a bridged connection through the VM why wouldnt the dockers works through the same setup? If your guide ''solves'' the NAT problem for the dockers?You say the second network has a NAT too, but why isn't that an issue if i use just 1 Instance of the miner without the dockers?Is it because the UAM resolves the NAT mapping ports through the Upnp only once? So basically your guide adds a virtual Upnp in the linux enviroment and the UAM maps through it natively just as it maps through my router NAT with his Upnp? So basically if the UAM goes thorugh the Linux NAT enviroment it can't get through my Router NAT enviroment again?

This question is already beyond the scope of this topic.
It describes how to mine on dedicated servers, and I will not analyze your particular case.
I already wrote above that you will have to solve the problem with double nat yourself.

Last edited by Face/Off (2022-01-09 23:35:24)

money_ranger · 2022-01-12 13:37:02

oh well thanks anyway then

kaiserdj · 2022-02-10 16:11:38

Could part 2 of the tutorial (Installing and configuring a Linux system for the Utopia network node) be encapsulated in a docker container and run from a docker network?

Would it be possible?

bumblebee · 2022-02-13 15:29:10

Superb how-to, the best I've seen in ages!

Followed to the dot, all worked 100% - installed portainer to see all behaving superbly and civilised - nice feature of the portainer is to look the logs of the non-gui instance - magic! - in addition to the stats.

Did not get greedy, but tried to push the number of the instances to the memory limits until it started swapping and things got interesting.

Hetzner bot eventually sent me a Network abuse notification.

Interesting, would one know why would one (or many) UAM instances started suddenly portscan a specific address, in my case 169.254.172.2, which is outside of the scope of the iptables rules?

Cudos to the author - you remind me of my early bitcoin days when I wrote and published a simple script to configure and run a headless GPU server that got downloaded over 10k times back in 2010.

bumblebee · 2022-02-15 11:57:06

bumblebee;1546 wrote:

...Interesting, would one know why would one (or many) UAM instances started suddenly portscan a specific address, in my case 169.254.172.2, which is outside of the scope of the iptables rules?

I know it is not cool to reply to own post, but this is to share how I resolved it.

It appears that the uam app (not sure why, there are no logs) is constantly portscanning this particular address which falls within the APIPA range of IP addesses while starting (and the miner goes on and off while it does that) so it ends up annoying the network routers.

To fix the network issue I just added a new line to the iptables.v4:

-A DOCKER-ISOLATION-STAGE-1 -d 169.254.0.0/16 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable

and then reloaded the iptables.v4 as described by the OP.

It is amazing to see how much APIPA traffic is rejected by the netfilter while the container starts, before processing its first block (+/- 45 minutes!)

Now, to solve another mystery - why some containers portscan that APIPA IP range and some have no problem and process blocks straight away - maybe it has to do with the network resources on my server???

Cromanes · 2022-02-24 17:39:14

Можно перевод инструкций на русский язык?

Can the instructions be translated into Russian?

Last edited by Cromanes (2022-02-24 17:39:35)

Face/Off · 2022-03-16 16:26:07

Cromanes;1755 wrote:

Можно перевод инструкций на русский язык?
Can the instructions be translated into Russian?

А, собственно, зачем? Там очень простой гайд, по большому счету все можно сделать копипастой команд, что большинство и делает.

Mine4Me · 2022-05-01 12:26:31

Hi,

First of all, thanks for you dedication onto this.

We use to have a similar way to accomplish same thing using docker, but now, we seem not to be able to run multiple miners on same machine, neither using your system, or ours.

We even tried to get a 64GB 8 cores at Hetzner (as suggested) and make it work... but as soon as you start over 3 - 4 instances of docker, they dockers simply looses all the incoming and out going connections, and does not work. It work if we launch only one inside or outside a docker (it will take a while to start mining, about 30 min), but as soon as you start adding more, it simply does not work.

Any suggestions?

Kind regards

Face/Off · 2022-05-08 21:09:01

Don't be greedy
However, if you look at the ratio of the number of views to donations, hardly anyone will hear my words
For my guide, I received 4 donations, and a total of 16 CRP

Last edited by Face/Off (2022-05-08 21:16:22)

bicpter · 2022-11-07 02:01:34

hi how can i allocate resources for CONTAINER ID? Thanks

Face/Off · 2022-11-20 23:00:26

bicpter;1912 wrote:

hi how can i allocate resources for CONTAINER ID? Thanks

I rarely come here, if you need help, find me on the channel "Utopia RUS | Mining support" (channel ID: 9481B440B6182EC69CE9371FE90B1491 ), so you will get help much faster.

Last edited by Face/Off (2022-11-20 23:04:57)

Kelechi · 2024-04-29 21:57:01

It's a way to streamline the setup and management of mining operations. However, it's worth noting that the profitability of cryptocurrency mining can vary based on factors like the cost of electricity, hardware efficiency, and the value of the cryptocurrency being mined.

uTalk

#1 2021-09-10 15:38:51